Nothing Special   »   [go: up one dir, main page]

CN112631527A - Juypter notewood code remote storage method and device based on k8s multi-tenant - Google Patents

Juypter notewood code remote storage method and device based on k8s multi-tenant Download PDF

Info

Publication number
CN112631527A
CN112631527A CN202110019611.0A CN202110019611A CN112631527A CN 112631527 A CN112631527 A CN 112631527A CN 202110019611 A CN202110019611 A CN 202110019611A CN 112631527 A CN112631527 A CN 112631527A
Authority
CN
China
Prior art keywords
notewood
juypter
code
storage
file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN202110019611.0A
Other languages
Chinese (zh)
Inventor
蔡猛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Minglue Artificial Intelligence Group Co Ltd
Original Assignee
Shanghai Minglue Artificial Intelligence Group Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Minglue Artificial Intelligence Group Co Ltd filed Critical Shanghai Minglue Artificial Intelligence Group Co Ltd
Priority to CN202110019611.0A priority Critical patent/CN112631527A/en
Publication of CN112631527A publication Critical patent/CN112631527A/en
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/062Securing storage systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/20Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2056Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/30Authentication, i.e. establishing the identity or authorisation of security principals
    • G06F21/44Program or device authentication
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0604Improving or facilitating administration, e.g. storage management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601Interfaces specially adapted for storage systems
    • G06F3/0602Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0626Reducing size or complexity of storage systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Security & Cryptography (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Quality & Reliability (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application discloses a method and a device for remotely storing Juypter notewood codes based on k8s multi-tenants, wherein the method comprises the following steps: obtaining S3 a storage client instance; the S3 is used for storing the client instance, and the custom Contents Manager interface is realized based on the S3 protocol, so that the Juypter notewood storage interface is realized; and (5) function deployment. According to the method and the device, a scheme that the notewood code file is directly stored in the remote end is adopted, the file is directly stored in the remote end, and the data is independently stored without the notewood, so that the persistent storage of the data is realized.

Description

Juypter notewood code remote storage method and device based on k8s multi-tenant
Technical Field
The invention relates to the technical field of machine learning. More specifically, the invention relates to a method and a device for remotely storing Juypter notewood codes based on k8s multi-tenancy.
Background
Juypter notewood is a browser-side development environment widely applied in the field of machine learning. The common deployment modes of Juypter notewood can be divided into physical machine deployment and k8 s-based containerized deployment. In the k8s (kubernets, an open source container orchestration engine) environment, notewood containers are often used as model development and training environments.
In the k8s environment, the k8s container does not support data storage persistence by default, i.e. there is a risk of loss when the notewood container exits abnormally or exits unsolicited descendant data. Currently, data persistence storage can be implemented by mounting an external data volume. In this implementation manner, when a new user starts the Notebook environment, a new data volume needs to be opened and mounted, at this time, the Notebook environment may directly operate the data volume storage as a local storage environment, and the developed code data may also be directly stored in the data volume. Therefore, when an exception occurs in the container, the data is already stored in the external data volume, and the data can be stored persistently.
In the above implementation scheme, there are still disadvantages that frequent manual intervention operation is required if a new user needs to access or newly open a notewood, an operation flow is to newly create an external data volume, and a mount operation is performed when the notewood is started, which consumes manpower. In addition, if the same volume mount is used to implement the operation of repeatedly creating a new volume, there is a problem of data privacy. In a Notebook container using the same data volume, code data in the data volume is completely visible, so that user data isolation cannot be achieved, and meanwhile, the risk that multiple persons modify the same code file simultaneously exists, so that the data of the code file is damaged or the code running result is abnormal.
Disclosure of Invention
The embodiment of the application provides a Juypter notewood code remote storage method based on k8s multi-tenancy, so as to at least solve the problem of subjective factor influence in the related technology.
The invention provides a Juypter notewood code remote storage method based on k8s multi-tenant, comprising the following steps:
an instantiation step: obtaining S3 a storage client instance;
the interface implementation step: and the S3 storage client instance is used, the custom Contents Manager interface is realized based on the S3 protocol, and the Juypter notewood storage interface is further realized.
As a further improvement of the present invention, the step of instantiating specifically includes the steps of:
an information acquisition step: acquiring the authentication information stored in the step S3;
a path generation step: and filling in a corresponding content Manager S3 storage information field according to the authentication information, simultaneously acquiring a user name and an environment label in an environment variable in the Juypter notewood, and combining the user name and the environment label to generate a storage path name.
As a further improvement of the present invention, the authentication information includes but is not limited to: access key, secret key, endpoint, storage space information.
As a further improvement of the present invention, the authentication information may be obtained by introducing a config code into the Juypter notewood.
As a further improvement of the present invention, the interface implementation step specifically includes the following steps:
an initialization process: initializing said S3 storing a client instance;
a file writing step: after initialization, the method realizes the related interface of the Juypter notewood operation ipynb file, and writes a setup file.
As a further improvement of the present invention, the present invention further includes a function deployment step, wherein the function deployment step specifically includes the following steps:
a code deployment step: performing code deployment under the setup.py file directory, and writing the content of the config file into a jupyter _ notebook _ config.py file under the-/-jupyter directory;
a configuration step: creating a bucket in the configuration for the Juypter notewood, and checking to pass;
packaging: packaging Juypter notewood codes into docker mirror images, and distributing the docker mirror images to a proxy mirror image warehouse;
a container deployment step: and performing container deployment by using the docker image based on k8s, and adding the user name and the environment label when the image is deployed, wherein the user name and the environment label are consistent with the config code.
Based on the same invention idea, the invention also discloses a distal storage device of the Juypter noteebook code based on k8s multi-tenant based on any one of the Juypter noteebook code remote storage methods based on k8s multi-tenant disclosed by the invention creation,
the Juypter notewood code remote storage device based on the k8s multi-tenant comprises:
an instantiation module, which acquires S3 storage client instance;
and the interface implementation module is used for storing the client instance by using the S3, and implementing a custom Contents Manager interface based on an S3 protocol, thereby implementing a Juypter notewood storage interface.
As a further improvement of the present invention, the instantiation module specifically includes:
an information acquisition unit that acquires the authentication information stored in S3;
and the path generating unit is used for filling a corresponding content Manager S3 storage information field according to the authentication information, acquiring a user name and an environment label in an environment variable in the Juypter notewood, and combining the user name and the environment label to generate a storage path name.
As a further improvement of the present invention, the interface implementation module specifically includes:
an initialization unit, which initializes the S3 storage client instance;
and after initialization, the file writing unit realizes the related interface of the Juypter notewood operation ipynb file and writes a setup.
As a further improvement of the present invention, the present invention further includes a function deployment module, where the function deployment module specifically includes:
a code deployment unit, which deploys codes in the setup.py file directory and writes the config file content into a jupyter _ notebook _ config.py file in the-/. jupyter directory;
the configuration unit is used for creating a bucket in the configuration for the Juypter notewood and verifying the bucket to pass;
the packaging unit is used for packaging the Juypter notewood code into a docker mirror image and distributing the docker mirror image to a proxy mirror image warehouse;
and the container deployment unit is used for deploying the container by utilizing the docker image based on the k8s, and when the image is deployed, the user name and the environment label are added and are kept consistent with the config code.
Compared with the prior art, the invention has the following beneficial effects:
1. the method for remotely storing the Juypter notewood code based on the k8s multi-tenant is provided, the scheme that the notewood code file is directly remotely stored is adopted, and the data can be independently stored without the notewood because the file is directly stored in the remote storage, so that the persistent storage of the data is realized;
2. a plurality of notewood containers can share one S3 storage bucket (S3 storage bucket), so that redundant manual intervention is not needed, and the labor is saved;
3. under the same user, data storage paths can be distinguished according to environment tags, a user code space self-defined classification function is provided, and the work efficiency of code development and k8s operation and maintenance personnel in machine learning can be effectively improved;
4. the remote persistent storage and data security isolation level of the code can be controlled in a user-defined mode, the operation and maintenance cost of the container environment is saved, the development environment can be isolated according to the user and the code of the user-defined label, and data storage security and data isolation are guaranteed;
5. on the premise of providing a k8s management system, a developer can autonomously control a container development environment without intervention of operation and maintenance personnel, and a related authority control module is not required to be additionally arranged in a remote operation and maintenance system, so that the scheme is light in weight, stable and controllable.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a flowchart of an overall method for storing the Juypter notewood code remotely based on k8s multi-tenants according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating the overall process of step S10 disclosed in FIG. 1;
FIG. 3 is a flowchart illustrating the overall process of step S20 disclosed in FIG. 1;
FIG. 4 is a flowchart illustrating the overall process of step S30 disclosed in FIG. 1;
FIG. 5 is a structural framework diagram of a remote storage device of Juypter notewood code based on k8s multi-tenant provided in this embodiment;
fig. 6 is a block diagram of a computer device according to an embodiment of the present invention.
In the above figures:
10. instantiating a module; 20. an interface implementation module; 30. a function deployment module; 11. an information acquisition unit; 12. a path generation unit; 21. an initialization unit; 22. a file writing unit; 31. a code deployment unit; 32. a configuration unit; 33. a packing unit; 34. a container deployment unit; 80. a bus; 81. a processor; 82. a memory; 83. a communication interface.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.
It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference to the terms "first," "second," "third," and the like in this application merely distinguishes similar objects and is not to be construed as referring to a particular ordering of objects.
The present invention is described in detail with reference to the embodiments shown in the drawings, but it should be understood that these embodiments are not intended to limit the present invention, and those skilled in the art should understand that the functional, methodological, or structural equivalents of these embodiments or alternatives thereof fall within the scope of the present invention.
Before describing in detail the various embodiments of the present invention, the core inventive concepts of the present invention are summarized and described in detail by the following several embodiments.
The method and the system can realize the remote storage of the Juypter notewood code based on the k8s multi-tenant, thereby realizing the data persistence storage.
The first embodiment is as follows:
referring to fig. 1 to 4, the present example discloses a specific implementation of a k8s multi-tenant based Juypter notewood code remote storage method (hereinafter referred to as "method").
Before describing the flow of the method, the implementation principle of the present embodiment is first described. The Jupyter notewood is developed by adopting python language, the code file is stored in an ipynb file format, and the data is in json format essentially. When code development is carried out, the newly-built ipynb file can send out a file saving instruction at regular time through notewood to be saved, the default saving position of the file is local storage, and file saving can also be carried out manually. And when the file is saved, the code content and the file saving path are sent to the back-end API, and the back-end API finishes updating the file content and finishes saving the ipynb file.
Specifically, referring to fig. 1, the method disclosed in this embodiment mainly includes the following steps:
and S10, acquiring S3 to store the client instance.
Specifically, in some embodiments, referring to fig. 2, the step S10 specifically includes the following steps:
s11, acquiring the authentication information stored in S3;
s12, filling the corresponding content Manager S3 storage information field according to the authentication information, simultaneously obtaining the user name and the environment label in the environment variable in the Juypter notewood, and combining the user name and the environment label to generate the storage path name.
Specifically, in some of these embodiments, the information needed during the instantiation process is first obtained. The authentication information which is required to be stored in S3 at the time of client initialization is stored in S3, and includes access key, secret key, endpoint, storage space bucket information, and the like. The authentication information can be obtained by introducing config code into Juypter notewood, and can be directly used in type initialization.
Specifically, in some embodiments, when the notewood config code adds configuration information, corresponding Contents Manager (for example, Custom Contents Manager) is filled according to the S3 authentication information S3 to store information fields, that is, corresponding Custom Contents Manager S3 is filled to store information fields, and it is necessary to obtain a user name and an environment tag in an environment variable env in the notewood and combine the two information to generate a unique storage path name.
Specifically, in the storage of S3, when a notebox is newly created, a user name and an environment tag are added to an environment variable in a container, and when a file is stored, a sublist storage is performed according to the user name and the environment tag. At this time, the isolation work of the data is realized through the sub-directory storage. A plurality of Notebook containers can share one S3 storage bucket (S3 storage bucket), and unnecessary manual intervention is not needed.
Specifically, in some embodiments, the step S12 is a key to implement multi-tenant file isolation, and the operation and maintenance system k8S is required to support an environment variable function of adding a currently logged-in user and a user-defined notebox environment tag when creating a Jupyter notebox container, so as to obtain authentication information in the notebox config code.
And then executing the step S20, storing the client instance by using the S3, and realizing a custom Contents Manager interface based on the S3 protocol so as to further realize a Juypter notewood storage interface.
Specifically, in some embodiments, referring to fig. 3, step S20 specifically includes the following steps:
s21, initializing the S3 storage client instance;
and S22, after initialization, realizing the related interface of the Juypter notewood operation ipynb file, and writing a setup.
Specifically, in some embodiments, in the Juypter notewood implementation storage interface, a notewood Contents Manager interface needs to be implemented, a Custom Contents Manager (taking a Custom Contents Manager as an example) needs to be implemented based on the S3 protocol, and a client instance needs to be stored in the Custom Contents Manager interface in S3, where the instance can be used as a read-write delete file stored in S3, and client instance initialization needs to be performed when the type is initialized.
Specifically, after the initialization of the S3 client is completed according to the configuration information of S3, the writing interface code realizes the relevant interface of the notewood operation ipynb file, and the linux common directory processing command during the writing of the code is as follows: file list ls, file judge isfile, directory judge isdir, file move mv, file copy cp, file delete rm, directory create mkdir, file read, file write, and writenotebook.
Specifically, after the interface code is completed, a setup.
Specifically, in some embodiments, the interface code further includes S30, function deployment after completion.
Specifically, referring to fig. 4, the step S30 specifically includes the following steps:
s31, performing code deployment in the setup.py file directory, and writing the config file content into a jupyter _ notebook _ config.py file in the-/. jupyter directory;
s32, creating a bucket in the configuration for the Juypter notewood, and checking to pass;
s33, packaging Juypter notewood codes into docker mirror images, and distributing the docker mirror images to a proxy mirror image warehouse;
and S34, container deployment is carried out by utilizing the docker image based on k8S, and the user name and the environment label are added and kept consistent with the config code when the image is deployed.
Specifically, code deployment is performed by using pip in the setup directory, and the Juypter Notebook code is formed by merging the Juypter Notebook original code and the interface code in the step S20.
Specifically, step S34 deploys the container by using the packaged mirror image based on k8S, and adds the user name and the environment tag in the container ENV when deploying the mirror image, and keeps the same as those in the notebook config code.
According to the method for remotely storing the Juypter notewood code based on the k8s multi-tenant, the scheme that the notewood code file is directly remotely stored is adopted, and the data can be independently stored without the notewood because the file is directly stored in the remote storage, so that the persistent storage of the data is realized; a plurality of notewood containers can share one S3 storage bucket (S3 storage bucket), so that redundant manual intervention is not needed, and the labor is saved; under the same user, data storage paths can be distinguished according to environment tags, a user code space self-defined classification function is provided, and the work efficiency of code development and k8s operation and maintenance personnel in machine learning can be effectively improved; the remote persistent storage and data security isolation level of the code can be controlled in a user-defined mode, the operation and maintenance cost of the container environment is saved, the development environment can be isolated according to the user and the code of the user-defined label, and data storage security and data isolation are guaranteed; on the premise of providing a k8s management system, a developer can autonomously control a container development environment without intervention of operation and maintenance personnel, and a related authority control module is not required to be additionally arranged in a remote operation and maintenance system, so that the scheme is light in weight, stable and controllable.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Example two:
in combination with the method for remotely storing the Juypter noteebook code based on the k8s multi-tenant disclosed in the first embodiment, this embodiment discloses a specific implementation example of a Juypter noteebook code remote storage device (hereinafter referred to as "device") based on the k8s multi-tenant.
Referring to fig. 5, the apparatus includes:
the instantiation module 10, acquiring S3 to store the client instance;
the interface implementation module 20 uses the S3 storage client instance to implement a custom Contents Manager interface based on the S3 protocol, thereby implementing a Juypter notewood storage interface.
Specifically, in some embodiments, the instantiation module 10 specifically includes:
an information acquisition unit 11 that acquires the authentication information stored in S3;
and the path generating unit 12 is configured to fill in a corresponding content Manager S3 storage information field according to the authentication information, obtain a user name and an environment tag in an environment variable in the Juypter notewood, and combine the user name and the environment tag to generate a storage path name.
Specifically, in some embodiments, the interface implementation module 20 specifically includes:
an initialization unit 21, which initializes the S3 storage client instance;
the file writing unit 22 is initialized to implement the relevant interface of the Juypter notewood operation ipynb file, and writes a setup.
Specifically, in some embodiments, the system further includes a function deployment module 30, where the function deployment module 30 specifically includes:
a code deployment unit 31, which deploys codes in the setup.py file directory and writes the config file content into a jupyter _ notebook _ config.py file in the-/. jupyter directory;
a configuration unit 32, which creates a bucket in the configuration for the Juypter notewood, and checks that the bucket passes;
the packing unit 33 is used for packing the Juypter notewood code into a docker mirror image and distributing the docker mirror image to a proxy mirror image warehouse;
and the container deployment unit 34 performs container deployment by using the docker image based on k8s, and adds the user name and the environment tag when deploying the image, and keeps the user name and the environment tag consistent with the config code.
The device for remotely storing a Juypter noteebook code based on k8s multi-tenant disclosed in this embodiment and the technical solution of the same parts in the method for remotely storing a Juypter noteebook code based on k8s multi-tenant disclosed in the first embodiment are described in the first embodiment, and are not described herein again.
Example three:
referring to fig. 6, the present embodiment discloses an embodiment of a computer device. The computer device may comprise a processor 81 and a memory 82 in which computer program instructions are stored.
Specifically, the processor 81 may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.
Memory 82 may include, among other things, mass storage for data or instructions. By way of example, and not limitation, memory 82 may include a Hard Disk Drive (Hard Disk Drive, abbreviated to HDD), a floppy Disk Drive, a Solid State Drive (SSD), flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 82 may include removable or non-removable (or fixed) media, where appropriate. The memory 82 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 82 is a Non-Volatile (Non-Volatile) memory. In particular embodiments, Memory 82 includes Read-Only Memory (ROM) and Random Access Memory (RAM). The ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), Electrically rewritable ROM (EAROM), or FLASH Memory (FLASH), or a combination of two or more of these, where appropriate. The RAM may be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), where the DRAM may be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended data output Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), and the like.
The memory 82 may be used to store or cache various data files for processing and/or communication use, as well as possible computer program instructions executed by the processor 81.
The processor 81 reads and executes the computer program instructions stored in the memory 82 to implement any one of the aforementioned embodiments of the method for remotely storing the Juypter notewood code based on k8s multi-tenancy.
In some of these embodiments, the computer device may also include a communication interface 83 and a bus 80. As shown in fig. 6, the processor 81, the memory 82, and the communication interface 83 are connected via the bus 80 to complete communication therebetween.
The communication interface 83 is used for implementing communication between modules, devices, units and/or equipment in the embodiment of the present application. The communication port 83 may also be implemented with other components such as: the data communication is carried out among external equipment, image/data acquisition equipment, a database, external storage, an image/data processing workstation and the like.
Bus 80 includes hardware, software, or both to couple the components of the computer device to each other. Bus 80 includes, but is not limited to, at least one of the following: data Bus (Data Bus), Address Bus (Address Bus), Control Bus (Control Bus), Expansion Bus (Expansion Bus), and Local Bus (Local Bus). By way of example, and not limitation, Bus 80 may include an Accelerated Graphics Port (AGP) or other Graphics Bus, an Enhanced Industry Standard Architecture (EISA) Bus, a Front-Side Bus (FSB), a Hyper Transport (HT) Interconnect, an ISA (ISA) Bus, an InfiniBand (InfiniBand) Interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a microchannel Architecture (MCA) Bus, a PCI (Peripheral Component Interconnect) Bus, a PCI-Express (PCI-X) Bus, a Serial Advanced Technology Attachment (SATA) Bus, a Video Electronics Bus (audio Electronics Association), abbreviated VLB) bus or other suitable bus or a combination of two or more of these. Bus 80 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.
The computer device can implement the distal storage of the Juypter notewood code based on k8s multi-tenants, thereby implementing the method described in conjunction with FIG. 1.
In addition, in combination with the method for remotely storing the Juypter notewood code based on the k8s multi-tenant in the foregoing embodiment, the embodiment of the present application may provide a computer-readable storage medium to implement. The computer readable storage medium having stored thereon computer program instructions; when executed by a processor, the computer program instructions implement any one of the preceding embodiments of the method for remotely storing the Juypter notewood code based on the k8s multi-tenant.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
In summary, the beneficial effects of the invention are that a method for remotely storing the Juypter notewood code based on k8s multi-tenants is provided, a scheme of directly remotely storing the notewood code file is adopted, and since the file is directly stored in the remote storage, the data can be independently stored without the notewood, thereby realizing the persistent storage of the data; a plurality of notewood containers can share one S3 storage bucket (S3 storage bucket), so that redundant manual intervention is not needed, and the labor is saved; under the same user, data storage paths can be distinguished according to environment tags, a user code space self-defined classification function is provided, and the work efficiency of code development and k8s operation and maintenance personnel in machine learning can be effectively improved; the remote persistent storage and data security isolation level of the code can be controlled in a user-defined mode, the operation and maintenance cost of the container environment is saved, the development environment can be isolated according to the user and the code of the user-defined label, and data storage security and data isolation are guaranteed; on the premise of providing a k8s management system, a developer can autonomously control a container development environment without intervention of operation and maintenance personnel, and a related authority control module is not required to be additionally arranged in a remote operation and maintenance system, so that the scheme is light in weight, stable and controllable.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (10)

1. A method for remotely storing Juypter notewood codes based on k8s multi-tenants is characterized by comprising the following steps:
an instantiation step: obtaining S3 a storage client instance;
the interface implementation step: and the S3 storage client instance is used, the custom Contents Manager interface is realized based on the S3 protocol, and the Juypter notewood storage interface is further realized.
2. The k8s multi-tenant-based Juypter notewood code remote storage method as claimed in claim 1, wherein the instantiating step specifically includes the steps of:
an information acquisition step: acquiring the authentication information stored in the step S3;
a path generation step: and filling in a corresponding content Manager S3 storage information field according to the authentication information, simultaneously acquiring a user name and an environment label in an environment variable in the Juypter notewood, and combining the user name and the environment label to generate a storage path name.
3. The k8s multi-tenant based Juypter notewood code remote storage method of claim 2, wherein the authentication information includes but is not limited to: access key, secret key, endpoint, storage space information.
4. The k8s multi-tenant-based Juypter noteebook code remote storage method as claimed in claim 3, wherein the authentication information is obtained by introducing config code into the Juypter noteebook.
5. The method for remotely storing the Juypter notewood code based on the k8s multi-tenant as claimed in claim 1, wherein the interface implementation step specifically includes the steps of:
an initialization process: initializing said S3 storing a client instance;
a file writing step: after initialization, the method realizes the related interface of the Juypter notewood operation ipynb file, and writes a setup file.
6. The method for remotely storing the Juypter notewood code based on the k8s multi-tenant as claimed in claim 1, further comprising a function deployment step, wherein the function deployment step specifically comprises the following steps:
a code deployment step: performing code deployment under the setup.py file directory, and writing the content of the config file into a jupyter _ notebook _ config.py file under the-/-jupyter directory;
a configuration step: creating a bucket in the configuration for the Juypter notewood, and checking to pass;
packaging: packaging Juypter notewood codes into docker mirror images, and distributing the docker mirror images to a proxy mirror image warehouse;
a container deployment step: and performing container deployment by using the docker image based on k8s, and adding the user name and the environment label when the image is deployed, wherein the user name and the environment label are consistent with the config code.
7. A distal storage device of Juypter notewood code based on k8s multi-tenant is characterized by comprising:
an instantiation module, which acquires S3 storage client instance;
and the interface implementation module is used for storing the client instance by using the S3, and implementing a custom Contents Manager interface based on an S3 protocol, thereby implementing a Juypter notewood storage interface.
8. The k8s multi-tenant-based Juypter notewood code remote storage of claim 7, wherein the instantiation module specifically comprises:
an information acquisition unit that acquires the authentication information stored in S3;
and the path generating unit is used for filling a corresponding content Manager S3 storage information field according to the authentication information, acquiring a user name and an environment label in an environment variable in the Juypter notewood, and combining the user name and the environment label to generate a storage path name.
9. The k8s multi-tenant-based Juypter notewood code remote storage device as claimed in claim 7, wherein the interface implementation module specifically includes:
an initialization unit, which initializes the S3 storage client instance;
and after initialization, the file writing unit realizes the related interface of the Juypter notewood operation ipynb file and writes a setup.
10. The k8s multi-tenant-based Juypter notewood code remote storage device as claimed in claim 7, further comprising a function deployment module, wherein the function deployment module specifically comprises:
a code deployment unit, which deploys codes in the setup.py file directory and writes the config file content into a jupyter _ notebook _ config.py file in the-/. jupyter directory;
the configuration unit is used for creating a bucket in the configuration for the Juypter notewood and verifying the bucket to pass;
the packaging unit is used for packaging the Juypter notewood code into a docker mirror image and distributing the docker mirror image to a proxy mirror image warehouse;
and the container deployment unit is used for deploying the container by utilizing the docker image based on the k8s, and when the image is deployed, the user name and the environment label are added and are kept consistent with the config code.
CN202110019611.0A 2021-01-07 2021-01-07 Juypter notewood code remote storage method and device based on k8s multi-tenant Withdrawn CN112631527A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110019611.0A CN112631527A (en) 2021-01-07 2021-01-07 Juypter notewood code remote storage method and device based on k8s multi-tenant

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110019611.0A CN112631527A (en) 2021-01-07 2021-01-07 Juypter notewood code remote storage method and device based on k8s multi-tenant

Publications (1)

Publication Number Publication Date
CN112631527A true CN112631527A (en) 2021-04-09

Family

ID=75291125

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110019611.0A Withdrawn CN112631527A (en) 2021-01-07 2021-01-07 Juypter notewood code remote storage method and device based on k8s multi-tenant

Country Status (1)

Country Link
CN (1) CN112631527A (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103036856A (en) * 2011-10-09 2013-04-10 镇江金软计算机科技有限责任公司 Multi-tenant system achievement based on software as a service (SAAS) application
US20140215452A1 (en) * 2013-01-28 2014-07-31 Red Hat, Inc. Deployment Optimization for High Availability in a Multi-Tenant Platform-as-a-Service (PaaS) System
CN105912691A (en) * 2016-04-21 2016-08-31 国电南瑞科技股份有限公司 Power grid model multi-version multi-tenant management system and method based on distributed storage
CN108833535A (en) * 2018-06-14 2018-11-16 浙江远算云计算有限公司 User data storage method based on the storage of cloud platform distributed block
CN109104467A (en) * 2018-07-25 2018-12-28 北京京东尚科信息技术有限公司 Develop environment construction method, apparatus and plateform system and storage medium
CN109740765A (en) * 2019-01-31 2019-05-10 成都品果科技有限公司 A kind of machine learning system building method based on Amazon server
CN110308994A (en) * 2014-02-19 2019-10-08 斯诺弗雷克公司 Resource management system and method
CN110765489A (en) * 2019-10-30 2020-02-07 深圳前海环融联易信息科技服务有限公司 Multi-tenant database isolation method and system, electronic device and computer storage medium
CN112035213A (en) * 2020-08-28 2020-12-04 北京白龙马云行科技有限公司 Multi-tenant network car booking system and dynamic isolation method

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103036856A (en) * 2011-10-09 2013-04-10 镇江金软计算机科技有限责任公司 Multi-tenant system achievement based on software as a service (SAAS) application
US20140215452A1 (en) * 2013-01-28 2014-07-31 Red Hat, Inc. Deployment Optimization for High Availability in a Multi-Tenant Platform-as-a-Service (PaaS) System
CN110308994A (en) * 2014-02-19 2019-10-08 斯诺弗雷克公司 Resource management system and method
CN105912691A (en) * 2016-04-21 2016-08-31 国电南瑞科技股份有限公司 Power grid model multi-version multi-tenant management system and method based on distributed storage
CN108833535A (en) * 2018-06-14 2018-11-16 浙江远算云计算有限公司 User data storage method based on the storage of cloud platform distributed block
CN109104467A (en) * 2018-07-25 2018-12-28 北京京东尚科信息技术有限公司 Develop environment construction method, apparatus and plateform system and storage medium
CN109740765A (en) * 2019-01-31 2019-05-10 成都品果科技有限公司 A kind of machine learning system building method based on Amazon server
CN110765489A (en) * 2019-10-30 2020-02-07 深圳前海环融联易信息科技服务有限公司 Multi-tenant database isolation method and system, electronic device and computer storage medium
CN112035213A (en) * 2020-08-28 2020-12-04 北京白龙马云行科技有限公司 Multi-tenant network car booking system and dynamic isolation method

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
叶振岳等: "基于多租户模式的节能检测及监控系统", 《计算机工程与科学》 *
龚华伟等: "基于S3语义的嵌入式在线存储网关设计", 《北京信息科技大学学报(自然科学版)》 *

Similar Documents

Publication Publication Date Title
US10341251B2 (en) Method and system for securely transmitting volumes into cloud
WO2017058577A1 (en) Nand-based verified boot
CN112395039B (en) Method and device for managing Kubernetes cluster
CN112541166B (en) Method, system and computer readable storage medium
CN111897623B (en) Cluster management method, device, equipment and storage medium
CN110365701B (en) Client terminal equipment management method and device, computing equipment and storage medium
US10585785B2 (en) Preservation of modifications after overlay removal from a container
US11775576B2 (en) Container image size reduction via runtime analysis
TWI737172B (en) Computer system, computer program product and computer implement method for incremental decryption and integrity verification of a secure operating system image
CN110753119A (en) Application package deployment system based on K8s cluster
CA3117314A1 (en) Secure invocation of network security entities
Sudharsan et al. Ota-tinyml: over the air deployment of tinyml models and execution on iot devices
CN112667415A (en) Data calling method and device, readable storage medium and electronic equipment
US20240241766A1 (en) Techniques for managing edge device provisioning
US9912454B2 (en) Systems and methods for efficient file transfer in a boot mode of a basic input/output system
CN107291486A (en) The installation method and device of a kind of operating system
CN110083366B (en) Application running environment generation method and device, computing equipment and storage medium
CN112631527A (en) Juypter notewood code remote storage method and device based on k8s multi-tenant
KR20210094883A (en) Method and system for dinamic application of storage encryption
CN115774742A (en) Data storage newly-increased method, device, equipment, medium and product of private cloud
CN114911421B (en) Data storage method, system, device and storage medium based on CSI plug-in
US10572166B1 (en) Firmware download for a solid state storage card
CN107533436B (en) Hardware management
CN111177752B (en) Credible file storage method, device and equipment based on static measurement
US10747710B2 (en) Dynamic association of application workload tiers to infrastructure elements in a cloud computing environment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication
WW01 Invention patent application withdrawn after publication

Application publication date: 20210409