CN112631527A - Juypter notewood code remote storage method and device based on k8s multi-tenant - Google Patents
Juypter notewood code remote storage method and device based on k8s multi-tenant Download PDFInfo
- Publication number
- CN112631527A CN112631527A CN202110019611.0A CN202110019611A CN112631527A CN 112631527 A CN112631527 A CN 112631527A CN 202110019611 A CN202110019611 A CN 202110019611A CN 112631527 A CN112631527 A CN 112631527A
- Authority
- CN
- China
- Prior art keywords
- notewood
- juypter
- code
- storage
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/062—Securing storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/44—Program or device authentication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0626—Reducing size or complexity of storage systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method and a device for remotely storing Juypter notewood codes based on k8s multi-tenants, wherein the method comprises the following steps: obtaining S3 a storage client instance; the S3 is used for storing the client instance, and the custom Contents Manager interface is realized based on the S3 protocol, so that the Juypter notewood storage interface is realized; and (5) function deployment. According to the method and the device, a scheme that the notewood code file is directly stored in the remote end is adopted, the file is directly stored in the remote end, and the data is independently stored without the notewood, so that the persistent storage of the data is realized.
Description
Technical Field
The invention relates to the technical field of machine learning. More specifically, the invention relates to a method and a device for remotely storing Juypter notewood codes based on k8s multi-tenancy.
Background
Juypter notewood is a browser-side development environment widely applied in the field of machine learning. The common deployment modes of Juypter notewood can be divided into physical machine deployment and k8 s-based containerized deployment. In the k8s (kubernets, an open source container orchestration engine) environment, notewood containers are often used as model development and training environments.
In the k8s environment, the k8s container does not support data storage persistence by default, i.e. there is a risk of loss when the notewood container exits abnormally or exits unsolicited descendant data. Currently, data persistence storage can be implemented by mounting an external data volume. In this implementation manner, when a new user starts the Notebook environment, a new data volume needs to be opened and mounted, at this time, the Notebook environment may directly operate the data volume storage as a local storage environment, and the developed code data may also be directly stored in the data volume. Therefore, when an exception occurs in the container, the data is already stored in the external data volume, and the data can be stored persistently.
In the above implementation scheme, there are still disadvantages that frequent manual intervention operation is required if a new user needs to access or newly open a notewood, an operation flow is to newly create an external data volume, and a mount operation is performed when the notewood is started, which consumes manpower. In addition, if the same volume mount is used to implement the operation of repeatedly creating a new volume, there is a problem of data privacy. In a Notebook container using the same data volume, code data in the data volume is completely visible, so that user data isolation cannot be achieved, and meanwhile, the risk that multiple persons modify the same code file simultaneously exists, so that the data of the code file is damaged or the code running result is abnormal.
Disclosure of Invention
The embodiment of the application provides a Juypter notewood code remote storage method based on k8s multi-tenancy, so as to at least solve the problem of subjective factor influence in the related technology.
The invention provides a Juypter notewood code remote storage method based on k8s multi-tenant, comprising the following steps:
an instantiation step: obtaining S3 a storage client instance;
the interface implementation step: and the S3 storage client instance is used, the custom Contents Manager interface is realized based on the S3 protocol, and the Juypter notewood storage interface is further realized.
As a further improvement of the present invention, the step of instantiating specifically includes the steps of:
an information acquisition step: acquiring the authentication information stored in the step S3;
a path generation step: and filling in a corresponding content Manager S3 storage information field according to the authentication information, simultaneously acquiring a user name and an environment label in an environment variable in the Juypter notewood, and combining the user name and the environment label to generate a storage path name.
As a further improvement of the present invention, the authentication information includes but is not limited to: access key, secret key, endpoint, storage space information.
As a further improvement of the present invention, the authentication information may be obtained by introducing a config code into the Juypter notewood.
As a further improvement of the present invention, the interface implementation step specifically includes the following steps:
an initialization process: initializing said S3 storing a client instance;
a file writing step: after initialization, the method realizes the related interface of the Juypter notewood operation ipynb file, and writes a setup file.
As a further improvement of the present invention, the present invention further includes a function deployment step, wherein the function deployment step specifically includes the following steps:
a code deployment step: performing code deployment under the setup.py file directory, and writing the content of the config file into a jupyter _ notebook _ config.py file under the-/-jupyter directory;
a configuration step: creating a bucket in the configuration for the Juypter notewood, and checking to pass;
packaging: packaging Juypter notewood codes into docker mirror images, and distributing the docker mirror images to a proxy mirror image warehouse;
a container deployment step: and performing container deployment by using the docker image based on k8s, and adding the user name and the environment label when the image is deployed, wherein the user name and the environment label are consistent with the config code.
Based on the same invention idea, the invention also discloses a distal storage device of the Juypter noteebook code based on k8s multi-tenant based on any one of the Juypter noteebook code remote storage methods based on k8s multi-tenant disclosed by the invention creation,
the Juypter notewood code remote storage device based on the k8s multi-tenant comprises:
an instantiation module, which acquires S3 storage client instance;
and the interface implementation module is used for storing the client instance by using the S3, and implementing a custom Contents Manager interface based on an S3 protocol, thereby implementing a Juypter notewood storage interface.
As a further improvement of the present invention, the instantiation module specifically includes:
an information acquisition unit that acquires the authentication information stored in S3;
and the path generating unit is used for filling a corresponding content Manager S3 storage information field according to the authentication information, acquiring a user name and an environment label in an environment variable in the Juypter notewood, and combining the user name and the environment label to generate a storage path name.
As a further improvement of the present invention, the interface implementation module specifically includes:
an initialization unit, which initializes the S3 storage client instance;
and after initialization, the file writing unit realizes the related interface of the Juypter notewood operation ipynb file and writes a setup.
As a further improvement of the present invention, the present invention further includes a function deployment module, where the function deployment module specifically includes:
a code deployment unit, which deploys codes in the setup.py file directory and writes the config file content into a jupyter _ notebook _ config.py file in the-/. jupyter directory;
the configuration unit is used for creating a bucket in the configuration for the Juypter notewood and verifying the bucket to pass;
the packaging unit is used for packaging the Juypter notewood code into a docker mirror image and distributing the docker mirror image to a proxy mirror image warehouse;
and the container deployment unit is used for deploying the container by utilizing the docker image based on the k8s, and when the image is deployed, the user name and the environment label are added and are kept consistent with the config code.
Compared with the prior art, the invention has the following beneficial effects:
1. the method for remotely storing the Juypter notewood code based on the k8s multi-tenant is provided, the scheme that the notewood code file is directly remotely stored is adopted, and the data can be independently stored without the notewood because the file is directly stored in the remote storage, so that the persistent storage of the data is realized;
2. a plurality of notewood containers can share one S3 storage bucket (S3 storage bucket), so that redundant manual intervention is not needed, and the labor is saved;
3. under the same user, data storage paths can be distinguished according to environment tags, a user code space self-defined classification function is provided, and the work efficiency of code development and k8s operation and maintenance personnel in machine learning can be effectively improved;
4. the remote persistent storage and data security isolation level of the code can be controlled in a user-defined mode, the operation and maintenance cost of the container environment is saved, the development environment can be isolated according to the user and the code of the user-defined label, and data storage security and data isolation are guaranteed;
5. on the premise of providing a k8s management system, a developer can autonomously control a container development environment without intervention of operation and maintenance personnel, and a related authority control module is not required to be additionally arranged in a remote operation and maintenance system, so that the scheme is light in weight, stable and controllable.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a flowchart of an overall method for storing the Juypter notewood code remotely based on k8s multi-tenants according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating the overall process of step S10 disclosed in FIG. 1;
FIG. 3 is a flowchart illustrating the overall process of step S20 disclosed in FIG. 1;
FIG. 4 is a flowchart illustrating the overall process of step S30 disclosed in FIG. 1;
FIG. 5 is a structural framework diagram of a remote storage device of Juypter notewood code based on k8s multi-tenant provided in this embodiment;
fig. 6 is a block diagram of a computer device according to an embodiment of the present invention.
In the above figures:
10. instantiating a module; 20. an interface implementation module; 30. a function deployment module; 11. an information acquisition unit; 12. a path generation unit; 21. an initialization unit; 22. a file writing unit; 31. a code deployment unit; 32. a configuration unit; 33. a packing unit; 34. a container deployment unit; 80. a bus; 81. a processor; 82. a memory; 83. a communication interface.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.
It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference to the terms "first," "second," "third," and the like in this application merely distinguishes similar objects and is not to be construed as referring to a particular ordering of objects.
The present invention is described in detail with reference to the embodiments shown in the drawings, but it should be understood that these embodiments are not intended to limit the present invention, and those skilled in the art should understand that the functional, methodological, or structural equivalents of these embodiments or alternatives thereof fall within the scope of the present invention.
Before describing in detail the various embodiments of the present invention, the core inventive concepts of the present invention are summarized and described in detail by the following several embodiments.
The method and the system can realize the remote storage of the Juypter notewood code based on the k8s multi-tenant, thereby realizing the data persistence storage.
The first embodiment is as follows:
referring to fig. 1 to 4, the present example discloses a specific implementation of a k8s multi-tenant based Juypter notewood code remote storage method (hereinafter referred to as "method").
Before describing the flow of the method, the implementation principle of the present embodiment is first described. The Jupyter notewood is developed by adopting python language, the code file is stored in an ipynb file format, and the data is in json format essentially. When code development is carried out, the newly-built ipynb file can send out a file saving instruction at regular time through notewood to be saved, the default saving position of the file is local storage, and file saving can also be carried out manually. And when the file is saved, the code content and the file saving path are sent to the back-end API, and the back-end API finishes updating the file content and finishes saving the ipynb file.
Specifically, referring to fig. 1, the method disclosed in this embodiment mainly includes the following steps:
and S10, acquiring S3 to store the client instance.
Specifically, in some embodiments, referring to fig. 2, the step S10 specifically includes the following steps:
s11, acquiring the authentication information stored in S3;
s12, filling the corresponding content Manager S3 storage information field according to the authentication information, simultaneously obtaining the user name and the environment label in the environment variable in the Juypter notewood, and combining the user name and the environment label to generate the storage path name.
Specifically, in some of these embodiments, the information needed during the instantiation process is first obtained. The authentication information which is required to be stored in S3 at the time of client initialization is stored in S3, and includes access key, secret key, endpoint, storage space bucket information, and the like. The authentication information can be obtained by introducing config code into Juypter notewood, and can be directly used in type initialization.
Specifically, in some embodiments, when the notewood config code adds configuration information, corresponding Contents Manager (for example, Custom Contents Manager) is filled according to the S3 authentication information S3 to store information fields, that is, corresponding Custom Contents Manager S3 is filled to store information fields, and it is necessary to obtain a user name and an environment tag in an environment variable env in the notewood and combine the two information to generate a unique storage path name.
Specifically, in the storage of S3, when a notebox is newly created, a user name and an environment tag are added to an environment variable in a container, and when a file is stored, a sublist storage is performed according to the user name and the environment tag. At this time, the isolation work of the data is realized through the sub-directory storage. A plurality of Notebook containers can share one S3 storage bucket (S3 storage bucket), and unnecessary manual intervention is not needed.
Specifically, in some embodiments, the step S12 is a key to implement multi-tenant file isolation, and the operation and maintenance system k8S is required to support an environment variable function of adding a currently logged-in user and a user-defined notebox environment tag when creating a Jupyter notebox container, so as to obtain authentication information in the notebox config code.
And then executing the step S20, storing the client instance by using the S3, and realizing a custom Contents Manager interface based on the S3 protocol so as to further realize a Juypter notewood storage interface.
Specifically, in some embodiments, referring to fig. 3, step S20 specifically includes the following steps:
s21, initializing the S3 storage client instance;
and S22, after initialization, realizing the related interface of the Juypter notewood operation ipynb file, and writing a setup.
Specifically, in some embodiments, in the Juypter notewood implementation storage interface, a notewood Contents Manager interface needs to be implemented, a Custom Contents Manager (taking a Custom Contents Manager as an example) needs to be implemented based on the S3 protocol, and a client instance needs to be stored in the Custom Contents Manager interface in S3, where the instance can be used as a read-write delete file stored in S3, and client instance initialization needs to be performed when the type is initialized.
Specifically, after the initialization of the S3 client is completed according to the configuration information of S3, the writing interface code realizes the relevant interface of the notewood operation ipynb file, and the linux common directory processing command during the writing of the code is as follows: file list ls, file judge isfile, directory judge isdir, file move mv, file copy cp, file delete rm, directory create mkdir, file read, file write, and writenotebook.
Specifically, after the interface code is completed, a setup.
Specifically, in some embodiments, the interface code further includes S30, function deployment after completion.
Specifically, referring to fig. 4, the step S30 specifically includes the following steps:
s31, performing code deployment in the setup.py file directory, and writing the config file content into a jupyter _ notebook _ config.py file in the-/. jupyter directory;
s32, creating a bucket in the configuration for the Juypter notewood, and checking to pass;
s33, packaging Juypter notewood codes into docker mirror images, and distributing the docker mirror images to a proxy mirror image warehouse;
and S34, container deployment is carried out by utilizing the docker image based on k8S, and the user name and the environment label are added and kept consistent with the config code when the image is deployed.
Specifically, code deployment is performed by using pip in the setup directory, and the Juypter Notebook code is formed by merging the Juypter Notebook original code and the interface code in the step S20.
Specifically, step S34 deploys the container by using the packaged mirror image based on k8S, and adds the user name and the environment tag in the container ENV when deploying the mirror image, and keeps the same as those in the notebook config code.
According to the method for remotely storing the Juypter notewood code based on the k8s multi-tenant, the scheme that the notewood code file is directly remotely stored is adopted, and the data can be independently stored without the notewood because the file is directly stored in the remote storage, so that the persistent storage of the data is realized; a plurality of notewood containers can share one S3 storage bucket (S3 storage bucket), so that redundant manual intervention is not needed, and the labor is saved; under the same user, data storage paths can be distinguished according to environment tags, a user code space self-defined classification function is provided, and the work efficiency of code development and k8s operation and maintenance personnel in machine learning can be effectively improved; the remote persistent storage and data security isolation level of the code can be controlled in a user-defined mode, the operation and maintenance cost of the container environment is saved, the development environment can be isolated according to the user and the code of the user-defined label, and data storage security and data isolation are guaranteed; on the premise of providing a k8s management system, a developer can autonomously control a container development environment without intervention of operation and maintenance personnel, and a related authority control module is not required to be additionally arranged in a remote operation and maintenance system, so that the scheme is light in weight, stable and controllable.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Example two:
in combination with the method for remotely storing the Juypter noteebook code based on the k8s multi-tenant disclosed in the first embodiment, this embodiment discloses a specific implementation example of a Juypter noteebook code remote storage device (hereinafter referred to as "device") based on the k8s multi-tenant.
Referring to fig. 5, the apparatus includes:
the instantiation module 10, acquiring S3 to store the client instance;
the interface implementation module 20 uses the S3 storage client instance to implement a custom Contents Manager interface based on the S3 protocol, thereby implementing a Juypter notewood storage interface.
Specifically, in some embodiments, the instantiation module 10 specifically includes:
an information acquisition unit 11 that acquires the authentication information stored in S3;
and the path generating unit 12 is configured to fill in a corresponding content Manager S3 storage information field according to the authentication information, obtain a user name and an environment tag in an environment variable in the Juypter notewood, and combine the user name and the environment tag to generate a storage path name.
Specifically, in some embodiments, the interface implementation module 20 specifically includes:
an initialization unit 21, which initializes the S3 storage client instance;
the file writing unit 22 is initialized to implement the relevant interface of the Juypter notewood operation ipynb file, and writes a setup.
Specifically, in some embodiments, the system further includes a function deployment module 30, where the function deployment module 30 specifically includes:
a code deployment unit 31, which deploys codes in the setup.py file directory and writes the config file content into a jupyter _ notebook _ config.py file in the-/. jupyter directory;
a configuration unit 32, which creates a bucket in the configuration for the Juypter notewood, and checks that the bucket passes;
the packing unit 33 is used for packing the Juypter notewood code into a docker mirror image and distributing the docker mirror image to a proxy mirror image warehouse;
and the container deployment unit 34 performs container deployment by using the docker image based on k8s, and adds the user name and the environment tag when deploying the image, and keeps the user name and the environment tag consistent with the config code.
The device for remotely storing a Juypter noteebook code based on k8s multi-tenant disclosed in this embodiment and the technical solution of the same parts in the method for remotely storing a Juypter noteebook code based on k8s multi-tenant disclosed in the first embodiment are described in the first embodiment, and are not described herein again.
Example three:
referring to fig. 6, the present embodiment discloses an embodiment of a computer device. The computer device may comprise a processor 81 and a memory 82 in which computer program instructions are stored.
Specifically, the processor 81 may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.
The memory 82 may be used to store or cache various data files for processing and/or communication use, as well as possible computer program instructions executed by the processor 81.
The processor 81 reads and executes the computer program instructions stored in the memory 82 to implement any one of the aforementioned embodiments of the method for remotely storing the Juypter notewood code based on k8s multi-tenancy.
In some of these embodiments, the computer device may also include a communication interface 83 and a bus 80. As shown in fig. 6, the processor 81, the memory 82, and the communication interface 83 are connected via the bus 80 to complete communication therebetween.
The communication interface 83 is used for implementing communication between modules, devices, units and/or equipment in the embodiment of the present application. The communication port 83 may also be implemented with other components such as: the data communication is carried out among external equipment, image/data acquisition equipment, a database, external storage, an image/data processing workstation and the like.
The computer device can implement the distal storage of the Juypter notewood code based on k8s multi-tenants, thereby implementing the method described in conjunction with FIG. 1.
In addition, in combination with the method for remotely storing the Juypter notewood code based on the k8s multi-tenant in the foregoing embodiment, the embodiment of the present application may provide a computer-readable storage medium to implement. The computer readable storage medium having stored thereon computer program instructions; when executed by a processor, the computer program instructions implement any one of the preceding embodiments of the method for remotely storing the Juypter notewood code based on the k8s multi-tenant.
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
In summary, the beneficial effects of the invention are that a method for remotely storing the Juypter notewood code based on k8s multi-tenants is provided, a scheme of directly remotely storing the notewood code file is adopted, and since the file is directly stored in the remote storage, the data can be independently stored without the notewood, thereby realizing the persistent storage of the data; a plurality of notewood containers can share one S3 storage bucket (S3 storage bucket), so that redundant manual intervention is not needed, and the labor is saved; under the same user, data storage paths can be distinguished according to environment tags, a user code space self-defined classification function is provided, and the work efficiency of code development and k8s operation and maintenance personnel in machine learning can be effectively improved; the remote persistent storage and data security isolation level of the code can be controlled in a user-defined mode, the operation and maintenance cost of the container environment is saved, the development environment can be isolated according to the user and the code of the user-defined label, and data storage security and data isolation are guaranteed; on the premise of providing a k8s management system, a developer can autonomously control a container development environment without intervention of operation and maintenance personnel, and a related authority control module is not required to be additionally arranged in a remote operation and maintenance system, so that the scheme is light in weight, stable and controllable.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method for remotely storing Juypter notewood codes based on k8s multi-tenants is characterized by comprising the following steps:
an instantiation step: obtaining S3 a storage client instance;
the interface implementation step: and the S3 storage client instance is used, the custom Contents Manager interface is realized based on the S3 protocol, and the Juypter notewood storage interface is further realized.
2. The k8s multi-tenant-based Juypter notewood code remote storage method as claimed in claim 1, wherein the instantiating step specifically includes the steps of:
an information acquisition step: acquiring the authentication information stored in the step S3;
a path generation step: and filling in a corresponding content Manager S3 storage information field according to the authentication information, simultaneously acquiring a user name and an environment label in an environment variable in the Juypter notewood, and combining the user name and the environment label to generate a storage path name.
3. The k8s multi-tenant based Juypter notewood code remote storage method of claim 2, wherein the authentication information includes but is not limited to: access key, secret key, endpoint, storage space information.
4. The k8s multi-tenant-based Juypter noteebook code remote storage method as claimed in claim 3, wherein the authentication information is obtained by introducing config code into the Juypter noteebook.
5. The method for remotely storing the Juypter notewood code based on the k8s multi-tenant as claimed in claim 1, wherein the interface implementation step specifically includes the steps of:
an initialization process: initializing said S3 storing a client instance;
a file writing step: after initialization, the method realizes the related interface of the Juypter notewood operation ipynb file, and writes a setup file.
6. The method for remotely storing the Juypter notewood code based on the k8s multi-tenant as claimed in claim 1, further comprising a function deployment step, wherein the function deployment step specifically comprises the following steps:
a code deployment step: performing code deployment under the setup.py file directory, and writing the content of the config file into a jupyter _ notebook _ config.py file under the-/-jupyter directory;
a configuration step: creating a bucket in the configuration for the Juypter notewood, and checking to pass;
packaging: packaging Juypter notewood codes into docker mirror images, and distributing the docker mirror images to a proxy mirror image warehouse;
a container deployment step: and performing container deployment by using the docker image based on k8s, and adding the user name and the environment label when the image is deployed, wherein the user name and the environment label are consistent with the config code.
7. A distal storage device of Juypter notewood code based on k8s multi-tenant is characterized by comprising:
an instantiation module, which acquires S3 storage client instance;
and the interface implementation module is used for storing the client instance by using the S3, and implementing a custom Contents Manager interface based on an S3 protocol, thereby implementing a Juypter notewood storage interface.
8. The k8s multi-tenant-based Juypter notewood code remote storage of claim 7, wherein the instantiation module specifically comprises:
an information acquisition unit that acquires the authentication information stored in S3;
and the path generating unit is used for filling a corresponding content Manager S3 storage information field according to the authentication information, acquiring a user name and an environment label in an environment variable in the Juypter notewood, and combining the user name and the environment label to generate a storage path name.
9. The k8s multi-tenant-based Juypter notewood code remote storage device as claimed in claim 7, wherein the interface implementation module specifically includes:
an initialization unit, which initializes the S3 storage client instance;
and after initialization, the file writing unit realizes the related interface of the Juypter notewood operation ipynb file and writes a setup.
10. The k8s multi-tenant-based Juypter notewood code remote storage device as claimed in claim 7, further comprising a function deployment module, wherein the function deployment module specifically comprises:
a code deployment unit, which deploys codes in the setup.py file directory and writes the config file content into a jupyter _ notebook _ config.py file in the-/. jupyter directory;
the configuration unit is used for creating a bucket in the configuration for the Juypter notewood and verifying the bucket to pass;
the packaging unit is used for packaging the Juypter notewood code into a docker mirror image and distributing the docker mirror image to a proxy mirror image warehouse;
and the container deployment unit is used for deploying the container by utilizing the docker image based on the k8s, and when the image is deployed, the user name and the environment label are added and are kept consistent with the config code.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110019611.0A CN112631527A (en) | 2021-01-07 | 2021-01-07 | Juypter notewood code remote storage method and device based on k8s multi-tenant |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110019611.0A CN112631527A (en) | 2021-01-07 | 2021-01-07 | Juypter notewood code remote storage method and device based on k8s multi-tenant |
Publications (1)
Publication Number | Publication Date |
---|---|
CN112631527A true CN112631527A (en) | 2021-04-09 |
Family
ID=75291125
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110019611.0A Withdrawn CN112631527A (en) | 2021-01-07 | 2021-01-07 | Juypter notewood code remote storage method and device based on k8s multi-tenant |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112631527A (en) |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103036856A (en) * | 2011-10-09 | 2013-04-10 | 镇江金软计算机科技有限责任公司 | Multi-tenant system achievement based on software as a service (SAAS) application |
US20140215452A1 (en) * | 2013-01-28 | 2014-07-31 | Red Hat, Inc. | Deployment Optimization for High Availability in a Multi-Tenant Platform-as-a-Service (PaaS) System |
CN105912691A (en) * | 2016-04-21 | 2016-08-31 | 国电南瑞科技股份有限公司 | Power grid model multi-version multi-tenant management system and method based on distributed storage |
CN108833535A (en) * | 2018-06-14 | 2018-11-16 | 浙江远算云计算有限公司 | User data storage method based on the storage of cloud platform distributed block |
CN109104467A (en) * | 2018-07-25 | 2018-12-28 | 北京京东尚科信息技术有限公司 | Develop environment construction method, apparatus and plateform system and storage medium |
CN109740765A (en) * | 2019-01-31 | 2019-05-10 | 成都品果科技有限公司 | A kind of machine learning system building method based on Amazon server |
CN110308994A (en) * | 2014-02-19 | 2019-10-08 | 斯诺弗雷克公司 | Resource management system and method |
CN110765489A (en) * | 2019-10-30 | 2020-02-07 | 深圳前海环融联易信息科技服务有限公司 | Multi-tenant database isolation method and system, electronic device and computer storage medium |
CN112035213A (en) * | 2020-08-28 | 2020-12-04 | 北京白龙马云行科技有限公司 | Multi-tenant network car booking system and dynamic isolation method |
-
2021
- 2021-01-07 CN CN202110019611.0A patent/CN112631527A/en not_active Withdrawn
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103036856A (en) * | 2011-10-09 | 2013-04-10 | 镇江金软计算机科技有限责任公司 | Multi-tenant system achievement based on software as a service (SAAS) application |
US20140215452A1 (en) * | 2013-01-28 | 2014-07-31 | Red Hat, Inc. | Deployment Optimization for High Availability in a Multi-Tenant Platform-as-a-Service (PaaS) System |
CN110308994A (en) * | 2014-02-19 | 2019-10-08 | 斯诺弗雷克公司 | Resource management system and method |
CN105912691A (en) * | 2016-04-21 | 2016-08-31 | 国电南瑞科技股份有限公司 | Power grid model multi-version multi-tenant management system and method based on distributed storage |
CN108833535A (en) * | 2018-06-14 | 2018-11-16 | 浙江远算云计算有限公司 | User data storage method based on the storage of cloud platform distributed block |
CN109104467A (en) * | 2018-07-25 | 2018-12-28 | 北京京东尚科信息技术有限公司 | Develop environment construction method, apparatus and plateform system and storage medium |
CN109740765A (en) * | 2019-01-31 | 2019-05-10 | 成都品果科技有限公司 | A kind of machine learning system building method based on Amazon server |
CN110765489A (en) * | 2019-10-30 | 2020-02-07 | 深圳前海环融联易信息科技服务有限公司 | Multi-tenant database isolation method and system, electronic device and computer storage medium |
CN112035213A (en) * | 2020-08-28 | 2020-12-04 | 北京白龙马云行科技有限公司 | Multi-tenant network car booking system and dynamic isolation method |
Non-Patent Citations (2)
Title |
---|
叶振岳等: "基于多租户模式的节能检测及监控系统", 《计算机工程与科学》 * |
龚华伟等: "基于S3语义的嵌入式在线存储网关设计", 《北京信息科技大学学报(自然科学版)》 * |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10341251B2 (en) | Method and system for securely transmitting volumes into cloud | |
WO2017058577A1 (en) | Nand-based verified boot | |
CN112395039B (en) | Method and device for managing Kubernetes cluster | |
CN112541166B (en) | Method, system and computer readable storage medium | |
CN111897623B (en) | Cluster management method, device, equipment and storage medium | |
CN110365701B (en) | Client terminal equipment management method and device, computing equipment and storage medium | |
US10585785B2 (en) | Preservation of modifications after overlay removal from a container | |
US11775576B2 (en) | Container image size reduction via runtime analysis | |
TWI737172B (en) | Computer system, computer program product and computer implement method for incremental decryption and integrity verification of a secure operating system image | |
CN110753119A (en) | Application package deployment system based on K8s cluster | |
CA3117314A1 (en) | Secure invocation of network security entities | |
Sudharsan et al. | Ota-tinyml: over the air deployment of tinyml models and execution on iot devices | |
CN112667415A (en) | Data calling method and device, readable storage medium and electronic equipment | |
US20240241766A1 (en) | Techniques for managing edge device provisioning | |
US9912454B2 (en) | Systems and methods for efficient file transfer in a boot mode of a basic input/output system | |
CN107291486A (en) | The installation method and device of a kind of operating system | |
CN110083366B (en) | Application running environment generation method and device, computing equipment and storage medium | |
CN112631527A (en) | Juypter notewood code remote storage method and device based on k8s multi-tenant | |
KR20210094883A (en) | Method and system for dinamic application of storage encryption | |
CN115774742A (en) | Data storage newly-increased method, device, equipment, medium and product of private cloud | |
CN114911421B (en) | Data storage method, system, device and storage medium based on CSI plug-in | |
US10572166B1 (en) | Firmware download for a solid state storage card | |
CN107533436B (en) | Hardware management | |
CN111177752B (en) | Credible file storage method, device and equipment based on static measurement | |
US10747710B2 (en) | Dynamic association of application workload tiers to infrastructure elements in a cloud computing environment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
WW01 | Invention patent application withdrawn after publication | ||
WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210409 |