EA047005B1 - METHODS, MACHINE READABLE MEDIA AND SYSTEMS FOR CREATING, ORGANIZING, VIEWING AND CONNECTING ANNOTATIONS TO DOCUMENTS IN WEB BROWSERS - Google Patents

Description

Prerequisites for creating the invention

The World Wide Web (WWW), web browsers and browser extensions are well known. The WWW is a global source of information where resources such as web pages, pdf documents, images and videos are identified by unique Uniform Resource Locators (URLs), and web browsers are software applications designed to access information on the WWW. Additionally, browser extensions are pieces of software designed to enhance a browser's capabilities.

Already existing web browsers. It is also known that the functionality of a web browser can be modified either by modifying the web browser's programming code or by providing a web browser extension for those web browsers that support adding functionality in this manner.

As the popularity of the World Wide Web grows, people on average spend more time per day using the World Wide Web than in the past. As the average time spent on the World Wide Web has increased, people have become more likely to retrieve information they previously saw on the WWW, post and discuss pieces of text in web documents with friends and colleagues, refer to specific pieces of text or context in a web document, ask asking specific questions about parts of web documents, linking information in web documents, finding and organizing content and snippets for later use, assessing the value of information gathered from the crowd, and accessing other people's public comments on web documents and reviewing those comments in context.

To cope with these problems, people typically repeat searches to find content they've already accessed, bookmark web documents, rescan articles for pieces they remember, copy and paste pieces of text into emails or other documents to share with friends or colleagues, refer to sentences or paragraphs by web page number or location during a phone call, ask questions in the comments section of an article and refer to the desired text by cutting and pasting it into a comment, load a web page, or Copy passages into a personal word processor or spreadsheet. This creates inefficiency and wasted time for a large number of users. Moreover, the lack of tools to identify inaccuracies and disseminate this knowledge to other users allows inaccurate information to persist longer than it should. Providing methods, machine-readable media, and systems to handle these tasks more efficiently would be beneficial.

Brief description of the invention

It is an object of the invention to provide users with an annotation-enabled web browser for annotating web pages and documents so that users elsewhere on the Internet also using such an annotation-enabled web browser can identify questions, points of interest, statements that agree , and statements that disagree without reading the user's comment text or the highlighted annotation text. This mechanism increases the efficiency with which the user sorts huge amounts of information on the WWW.

Another object of the invention is to provide users with an annotation-enabled Web browser for annotating images, videos, and sound recordings uploaded or embedded in Web pages so that users elsewhere on the Internet also using such an annotation-enabled Web browser can view those images, videos and sound recordings with annotations.

Another object of the invention is to ensure that the viewing context of a document after adding annotations changes minimally compared to the viewing context of a document without annotations. Using a minimally modified document viewing context allows users to view content without being distracted by annotations, while still being able to find and use annotations when they need them.

Another object of the invention is to unobtrusively present advertising to users. One goal is to show advertising content to users after those users initiate a curiosity-driven action, where curiosity is triggered by presenting an advertisement in a specific location or context on a web page. This placement or context suggests that the advertisement may be relevant to users' current needs. Moreover, the presented ad without its content may contain some aspects of the full ad, but all the details are not displayed until the user initiates an action driven by curiosity, which is provoked by contextual methods that selectively and transparently target the user based on his current needs.

Another aspect of the invention is to provide a separate annotation contextual view area that is separate from the World Wide Web document contextual view area. This contextual annotation viewing area has a mechanism, in addition to searching and filtering, that presents annotations to users in such a way that users can understand the site of origin for familiar sites and the approximate location of the annotation. In addition, other elements

- 1 047005 ments of the context of an annotation or highlighted text can be made clear without having to read the highlighted text or the user's comment on the annotation. This mechanism makes it easier to find a specific annotation among a large number of other annotations.

Another aspect of the invention is to provide a contextual organization and annotation area that allows users to scan and organize annotations into meaningful groups. The benefit of this object is that it allows users to focus on creating annotations first, and once they are created, focus on organizing them. Additionally, the default annotation organization does not need to be overridden.

Another aspect of the invention is to provide users with a contextual annotation view area that allows the user to browse pages of annotations, save some annotations available for later viewing, and continue searching for other annotations. In addition, the method facilitates mental tracking by keeping annotations in the visual field of view and moving annotations of no interest out of the visual field of view.

A further object of the invention is the transparent transmission of privacy settings to users at any time. Sharing privacy settings will help avoid accidentally sharing information that the user does not want to share.

Another object of the invention is a method for creating tours of a number of websites on the Internet, drawing users' attention to the main points of interest on each site. This object also includes ways to modify the tour and present the tour to users.

Another object of the invention is to provide a way for groups to control access, control permissions, and communicate through messaging.

Brief description of several types of drawings

For a more complete understanding of the invention, reference is made to the following description and accompanying drawings, in which:

in fig. Figure 1 shows a flowchart for creating annotations, moving them into two different collections, and sharing them with other users through the collections;

in fig. Figure 2 shows a flowchart of annotation viewing on two separate web pages on two separate days;

in fig. 3 illustrates an annotation-enabled web browser in an exemplary embodiment in which the annotation-capable web browser has been augmented with an annotation-enabled web browser extension prior to the annotation process;

in fig. 4 shows a web browser with the ability to annotate after text selection;

in fig. 5 shows a web browser with the ability to annotate before selecting an annotation symbol;

in fig. 6 shows a web browser with annotation capability after selecting an annotation symbol;

in fig. 7 shows a web browser with annotation capability after an annotation has been created while the displayed annotation symbol is in the open state;

in fig. 8 shows a web browser with annotation capability after an annotation has been created and the displayed annotation symbol is in a closed state;

in fig. 9 shows a block of annotation elements that provides context for the annotation when it is presented in a separate context area and not presented as an overlay on an existing web document next to the selected annotation text;

in fig. 10 shows an annotation organization context area presenting fields for annotation elements and display collections of entries in a separate context area that is separate from the source web documents containing the highlighted annotation texts; in fig. 11 shows the annotation organization contextual area, demonstrating the process of moving and duplicating annotations in a collection, which are groups of annotations and other elements;

in fig. 12 shows an annotation-enabled web browser showing a web document with annotation symbols displayed in a closed state;

in fig. 13 illustrates an annotation-enabled web browser showing a web document with annotation symbols displayed in a closed state with corresponding annotation symbol beacons displayed;

in fig. 14 is a sequence diagram showing the relationship between a web server capable of annotation, a web server providing visited URL data, an annotation server, an electronic database and a CDN;

in fig. 15 is a sequence diagram showing the relationship between a web browser, an annotation server, an electronic database, and a CDN for organizing annotations; in fig. 16 is a sequence diagram showing the relationship between an annotation-enabled web browser, a web server providing visited URL data, an annotation server, and an electronic database;

in fig. 17 is a flowchart showing acquisition and storage of a data element packet;

- 2 047005 in Fig. 18 is a flowchart showing composition and transmission of a burst display set;

in fig. 19 is a flowchart showing the interaction of a user, a web browser, a web browser extension content script, and a web browser extension background script to allow a user to view a web page and create an annotation on the page; in fig. 20 is a flow diagram showing how a user, a web browser, a web browser extension background script, and a web browser extension content script interact to allow a user to view a web page with annotations on the page; in fig. 21 is a flow diagram showing how a nurse using this exemplary embodiment creates a glycemic index notebook to share with a patient;

in fig. 22 is a flow chart showing a patient using a notepad provided by a nurse;

in fig. 23 shows the notepad user interface as part of the extension; in fig. 24 is a diagram of the sequence of creating a notebook and creating and adding one annotation to the notebook;

in fig. 25 is a diagram of the sequence of viewing a notebook, clicking on the first annotation, moving the browser to that url, and displaying the annotation after scrolling to the appropriate location;

in fig. 26 is a flowchart showing creating a notebook and adding the already created annotation to the notebook;

in fig. 27 is a flowchart showing viewing a notebook and opening an annotation in the notebook;

in fig. 28 shows an approximate version of the system diagram;

in fig. 29 shows another exemplary component diagram for an annotation server and electronic database;

in fig. 30 is an exemplary embodiment showing client electronic devices for an annotation server;

in fig. 31 shows an example of a collection items page for the US History collection; in fig. 32 shows an example version of the collections page;

in fig. 33 is an example of an inbox page showing a system message;

in fig. 34 shows an example of an inbox page after a user has responded to a system message and other users have also sent messages;

in fig. Figure 35 shows the interface for viewing image annotations and creating an image annotation group;

in fig. 36 shows an image annotation group creation interface with one annotation group list item added to the candidate group list;

in fig. 37 shows an image annotation group creation interface with one annotation group list item in which the second image annotation without collision is not added to the candidate annotation group list;

in fig. 38 shows an image annotation group creation interface with two annotation group list items added to the candidate group list;

in fig. 39 shows an image annotation group creation interface with one annotation group list item added to the candidate group list, with a second colliding image annotation not added to the candidate group list;

in fig. 40 shows an interface for viewing video annotations and creating a group of video annotations;

in fig. 41 shows an interface for viewing video annotations and creating a video annotation group with one video annotation added to the annotation candidate group list window, with motion trajectories for video annotation elements;

in fig. 42 shows a video annotation group creation interface with two annotation group list items added to the motion path candidate group list for the second video annotation items;

in fig. 43 shows a video annotation group creation interface with one annotation group list item added to the annotation candidate group list, with a second colliding video annotation not added to the annotation candidate group list;

in fig. 44 shows an interface for viewing audio annotations and creating an audio annotation group with an audio annotation timeline, and an interface for creating a derivative audio annotation;

in fig. 45 shows an interface for viewing video annotations and creating a video annotation group with a video annotation timeline, and an interface for creating a derivative video annotation;

in fig. 46 shows an annotation-enabled web browser displaying a video with embedded beacons for annotating the video;

in fig. 47 shows a web browser with annotation capability after creating an image annotation, and the embedded displayed image annotation is in the open state;

- 3 047005 in Fig. 48 shows a web browser with annotation capability after creating a video annotation, and the embedded displayed video annotation is in the open state during the playback range of the video annotation; and in fig. 49 shows a web browser with annotation capability after the audio annotation is created, and the embedded displayed audio annotation is in the open state during the playback range of the audio annotation.

Detailed Description of the Invention

Methods, non-volatile machine-readable media (NEM) and systems are designed for annotating web pages, documents, images, videos and audio recordings in web browsers; organizing these annotations; viewing these annotations; and compiling these annotations into useful digital notebooks that include a set of annotations. A web document has a url and includes web pages, pdf documents, and other types of documents that are displayed in web browsers. In addition, these methods, non-volatile COs, and systems provide tools for curating content, sharing content with other users, and managing access control. The descriptions contain block diagrams that show how the user may use the systems and non-volatile computer readable media (NVEM) described in this document. Additionally, graphical user interface (GUI) diagrams show the end result of executing instructions on computer-readable media and what systems display. In addition, sequence diagrams show the interactions and data flow between different components, while flowcharts describe methods. In addition, floating diagrams show the actions of various components.

An annotation is an explanatory note, comment or drawing added to text, diagram, picture, document, video or sound recording. Describing methods, intransitive reference systems, and systems introduces terminology for describing annotation in various contexts, as well as data elements and structures for representing annotation and objects that add annotation context to web pages, web documents, documents, images, video, and audio, and into separate context areas.

In fig. Figure 1 shows a flowchart of a teacher's use of an example web browser with annotation capabilities. The teacher has two separate goals. The first goal is to create a list of study questions about presidents for students in the teacher's class. The teacher's class is small and students can see each other's answers in this example in the exemplary embodiment depicted.

The second goal is to find a house for sale near the school where the teacher teaches. As shown in FIG. 1, on the first day [1], the teacher visits Abraham Lincoln's Wikipedia page and creates a question annotation [4], and then visits Woodrow Wilson's Wikipedia page and creates a question annotation [5]. As shown in FIG. 1, On the second day [2], the teacher visits the Zillow page and creates a dagger annotation of a house for sale near work [6]. As shown in FIG. 1, on the tenth day [3], the teacher creates two collections called American Presidents Study Guide and Real Estate [7], adds students to the American Presidents Study Guide collection and adds a roommate to the Real Estate collection [8], moves the question annotations about Abraham Lincoln and Woodrow Wilson in the American Presidents Study Guide [9], moves dagger-like blurbs about a house for sale next to a job to the Real Estate collection [10], and shares the American Presidents Study Guide collection with students [11]. In fig. Figure 2 shows a flowchart of a student using an example web browser with annotation capabilities. The student's goal is to answer learning questions asked by the teacher. As shown in FIG. 2, on the eleventh day [12], the student visits the Abraham Lincoln Wikipedia page [14], reads the Abraham Lincoln Wikipedia page [15], sees the closed question annotation symbol displayed by the teacher while reading the Abraham Lincoln Wikipedia page [16], clicks on the closed symbol question annotation displayed by the teacher, causing the question annotation symbol to go into an open state and show the teacher's annotation comment [17], and respond to the teacher's question annotation comment with an answer [18]. As shown in FIG. 2, student on day twelve [13] visits Woodrow Wilson's Wikipedia page [19], reads Woodrow Wilson's Wikipedia page [20], sees closed question annotation symbol displayed by teacher while reading Woodrow Wilson's Wikipedia page [21], clicks on symbol closed question annotation displayed by the teacher, causing the question annotation symbol to go into an open state and show the teacher's annotation comment [22], and respond to the teacher's answer annotation comment [23].

In fig. Figure 3 shows an example implementation of a web browser with annotation capabilities [24]. presented. The short, wide horizontal rectangles that appear in the viewport of the web browser shown here represent text. In the first paragraph, 11 rectangles represent 11 lines of text, and in the second paragraph, 5 rectangles represent 5 lines of text. In this exemplary embodiment, a web browser that supports adding web extensions has been modified using an exemplary web browser extension [25] that adds annotation capabilities. A web browser extension [25] is a software package, usually located on a non-volatile computer-readable medium, that can be loaded into the memory of a web browser that supports adding web extensions when

- 4 047005 In this case, the web browser application itself is a software application loaded into memory and running on the processor of an electronic computing device. Once a web browser application loads the web browser extension [25] into memory, the web browser extension's instruction set [25] can be invoked by the web browser application's instruction set at certain points in the web browser's life cycle. Some of the effects of the web browser extension command set [25] on the user interface presented to the user are shown in FIG. 5, fig. 6, fig. 7, fig. 8, fig. 12, fig. 13, fig. 23, fig. 46, fig. 47, fig. 48 and fig. 49.

In fig. Figure 4 shows an exemplary implementation of a web browser [24] with the ability to annotate after the user selects text, called regular highlighted text [26], and before running a set of web browser extension commands [25].

In fig. Figure 5 shows an exemplary implementation of a web browser [24] with the ability to annotate after the user selects text and after a set of web browser extension commands [25] is executed to cause the annotation symbol selection button bar to appear [28]. Moreover, the highlighted text will be referred to as annotation highlighted text [27] to distinguish it from normal highlighted text [26], as shown in the state of the exemplary embodiment shown in FIG. 4. Regular highlighted text [26] is the selected text before the web browser extension command set [25] is triggered, or the selected text in a web browser that does not have annotation capabilities. The web browser extension command set [25] processes input after the user clicks any of the buttons depicted in the annotation symbol, selection button bar [28] in FIG. 5.

In one exemplary embodiment, FIG. 6 shows the effect of a user clicking the question mark annotation symbol button on the annotation symbol selection button bar [28] shown in FIG. 5. In an alternative exemplary embodiment, the annotation symbols shown in the annotation symbol selection button bar [28] in FIG. 5 may be arranged one above the other in a vertical column rather than in a horizontal row. In the exemplary embodiment shown in FIG. 6 shows the displayed annotation symbol [29] with an open question and the annotation comment input field [30]. The open-ended question annotation symbol [29] allows the user to indicate that they plan to enter a question about the selected annotation text [27] in the annotation comment input field [30]. The annotation comment input field [30] is an annotation-capable input area for the user of a web browser [24] to enter text to be stored by one or more computer systems. Once the user finishes entering text and passes the completion of the text entry action to the annotable web browser [24], the web browser extension command set [25] places the annotable web browser [24] into the state depicted in FIG. 7.

In the exemplary embodiment depicted in FIG. 7, the state of the web browser [24] with annotation capability is shown after creating an annotation, and the displayed annotation symbol is visually displayed in the open state. The annotation display symbol, when open, displays screen elements including highlighted annotation text [27], annotation comment [32], reputation selector [33], open question annotation display symbol [29], and annotation comment field [31]. An annotation comment [32] is generated from text entered by the user, as described previously. The annotation comment field [31] also contains a contextual variant annotation control panel [34]. A contextual annotation options control panel [34] allows the user, depending on access control rules, to change privacy settings, share on social networks or via email, mark the annotation, or delete it. The reputation selector [33] allows the user to select a reputation to assign to a comment annotation [32]. For example, a user can assign a reputation to an annotation comment [32] as Thoughtful if they believe that the comment was thoughtful. By clicking outside of the highlighted annotation text [27], the annotation display symbol [29] with an open question, and the annotation comment field [31], the user can cause the annotation display symbol to transition to a closed state, which closes the annotation. In an alternative implementation, the reputation selector [33] could potentially be displayed within the contextual option annotation control panel [34], rather than being located next to the annotation comment [32].

In the exemplary embodiment depicted in FIG. 8, the state of the web browser [24] with annotation capability is shown after creating an annotation and displaying the annotation symbol in the closed state. In fig. Figure 8 shows the annotation symbol [35] displayed with a closed question. The displayed closed question annotation symbol [35] is intended to be visible without distracting the user from the rest of the text.

After a user creates an annotation using a web browser [24] with annotation capability, a package of data elements representing the annotation is sent to the server. This set of data elements can be used to construct a block of annotation elements [36], in individual context regions, as shown in the example implementation depicted in FIG. 9. In the exemplary embodiment shown in FIG. 9, the annotation element block [36] contains a screenshot of the marked text [37], separately selected contextual annotation text [38], separately displayed

- 5 047005 contextual question annotation symbol [39], a separate contextual annotation comment [40], a separate control panel for the contextual annotation option [41], and a separate contextual reputation selector [42]. The Tagged Text Screenshot [37] in this exemplary embodiment is a thumbnail-sized screenshot of tagged text designed to allow a user to identify the web page on which the annotation is located and the location of the annotation on the web page. Fig. 7 and fig. 9 may be viewed as two different contexts of the same exemplary embodiment, or as two different exemplary embodiments. To understand the highlighted text of a single context annotation [38], the displayed symbol of a single context question [39], and the comment of a single context annotation [40], it is useful to consider these two figures as two different contexts of the same example implementation. Selected annotation text in separate context [38] represents highlighted annotation text [27] in a separate context area of the annotation element field [36]. The description of a separate context region in this situation indicates that the context is different from the original web page or web document on which the highlighted annotation text resides [27]. A question annotation symbol [39] displayed in a separate context represents a question annotation symbol in a separate context area of an annotation element field [36]. A separate contextual annotation comment [40] represents an annotation comment [32] in a separate contextual region of the annotation element field [36]. A separate contextual annotation control panel [41] is designed to allow the user to change privacy settings, share via social media or email, associate a reputation categorization with the annotation, and delete the annotation.

If a block of elements represents an annotation, it is called an annotation element block [36], which is illustrated in an exemplary embodiment in FIG. 9. In an exemplary embodiment of the invention, an element field may represent web pages, documents, images, videos, audio recordings, as well as annotations for web pages, documents, images, video and audio recordings. The element block may be a web page element block, a document element block, an image element block, a video element block, an audio element block, an annotation element block, an image annotation element block, a video annotation element block, an audio annotation element block, an inline annotation element block images, an inline video annotation element block, an inline audio element block, an annotation group element block, a derived annotation group element block, and an annotation group element block with derived annotations. Collections are groups of elements that include annotations, and writable collections are collections that the user can modify in some way. One form of modification is to add an element to a writable collection. Collections are used to organize elements that include annotations into meaningful groups. As an example, in FIG. 1 teacher created two collections, one entitled Study Guide on US Presidents and the other entitled Real Estate.

In the exemplary embodiment depicted in FIG. 9, the annotation element block [36] contains a separate contextual reputation selector [42] in a separate context variant annotation control panel [41], which allows users with whom the annotation is shared as an annotation element field [36] to vote for reputation by selecting it from the drop-down list. This reputation is assigned to the annotation. This reputation has value in the social community. Annotations with a reputation for being useful, funny, or thoughtful, as well as annotations that express disagreement with the highlighted annotation text, can be selectively displayed based on various criteria selected by the user and deemed useful by the annotation server [61], as shown in some other figures. Users can filter annotations based on this criterion, and a user's aggregate reputation can also be linked to the reputation of annotations created by the user. Unlike star-based voting, detailed, annotation-specific semantic information can be collected from users about why the annotation is worthy of attention from other users in the online community.

Additionally, the annotation system functions better when users can respond to annotations in addition to assigning a reputation to the annotation. Additionally, other users can vote on the answer's reputation. Such a system helps the community identify annotations that may be inaccurate as users provide feedback through replies and reputation.

As an exemplary embodiment shown in FIG. 7 and the exemplary version shown in FIG.

9 are annotations. In fig. 7 shows an annotation embedded in a web document that is being annotated, and FIG. 9 - annotation outside the web document. Since in FIG. 9 lacks the surrounding web document context to give users viewing the annotation element field [36] more context, a screenshot labeled with a thumbnail is presented. fig. 7 and fig. 9 may potentially depict two contexts of the same exemplary embodiment, or they may potentially depict two different exemplary embodiments. As in fig. 7, and in Fig. 9 voting by reputation is allowed. In fig. 7, the reputation selector [33] allows the user to assign a reputation to an annotation comment [32]. In fig. 9 separate contextual reputation selector

- 6 047005

[42] allows the user to assign a reputation to an annotated comment [32]. This allows users with whom the annotation is shared to evaluate the reputation of the annotation in different contexts.

In the exemplary embodiment depicted in FIG. 9, the annotation element window [36] contains a separate annotation control panel [41] that allows the user to change privacy settings. In the example, dark gray indicates the most private setting where no one other than the creator of the annotation can view the annotation, light gray indicates that the annotation can be shared with users in specific groups, and white indicates that the annotation can be viewed all users of the platform.

One of the individual context areas in which the block of annotation elements [36] shown in FIG. 9 is in the annotation organization context area [43] of the exemplary embodiment depicted in FIG. 10. Annotation organization context area [43] in FIG. 10 also contains an Item Bin [44] for displaying blocks of items and a Record Collection Bin [45] for displaying collections of entries. In fig. 9, an annotation element block [36] is depicted inside an element basket [44], and a displayed writable collection [50] is depicted inside a writable collection basket [45]. Additionally, the search filter [46] can be used to change the item slots displayed in the item bin [44]. Additionally, the collection search filter [47] can be used to change the display of recordable collections displayed in the recordable collections cart [45]. Additionally, the paging mechanism [48] can be used to change the cell page of the items displayed in the items cart [44], and the collection paging mechanism [49] can be used to change the collection page of the displayed writable collections displayed in the writable collections cart [45 ].

In the exemplary embodiment depicted in FIG. 10, we see the paging mechanism [48] in the item basket [44] and the collection paging mechanism [49] in the writable collection basket [45]. These paging mechanisms operate independently of each other. The paging mechanism [48] in the element basket [44] moves back and forth across the page of 6 elements in the exemplary embodiment shown. The elements shown in FIG. 10 are annotations. All these elements are sequentially ordered. When the paging mechanism is moved forward, the next 6 items are displayed in a sequentially ordered set of items. Moving the paging engine backwards displays the previous 6 elements in a sequentially ordered set of elements. The collection swap engine [49] in the writable collection bin [45] moves back and forth through a page of 3 collections in the example embodiment shown. The collections shown in FIG. 10 represent a group of elements including annotations. Each collection represents a group of one or more elements, and the elements include web page elements, document elements, image elements, video elements, audio elements, annotation elements, image annotation elements, video annotation elements, audio annotation elements, inline image annotation elements, embedded video annotation elements, embedded audio elements, annotation group elements, derived annotation group elements, and an annotation group with derived annotation elements. All these collections are sequentially ordered. Moving the paging mechanism forward displays the next 3 items in a sequentially ordered set of collections. Moving the paging mechanism backwards displays the previous 3 items in a sequentially ordered set of collections. As the browser viewport changes after resizing operations, the number of displayed items in the items bin [44] and displayed writable collections in the writable collections bin [45] changes. In an alternative exemplary implementation, there may be no explicit write permission. In this case, the recycle bin [45] for recorded collections will include the displayed collections. The paging engine [48] and collection paging engine [49] take this change into account and then work with the new page sizes.

In fig. In Fig. 10 we also see pins in each annotation element block [36], located in the element basket [44] in addition to the paging mechanism [48]. In the exemplary embodiment shown in FIG. 10 shows 6 elements representing annotations. Each of these 6 elements can be in a loose or secured state. As shown, the page size is 6. If one of these items is pinned, the page size is reduced to 5. Pinning causes the pinned item to remain in the viewport when the paging mechanism [48] is moved forward or backward. When the paging engine [48] moves forward, the pinned element remains in the viewport, and the remaining 5 elements are removed from the viewport, viewport. Then the next 5 elements in the sequential element ordering replace the 5 elements removed. If the paging engine is moved backwards, the 5 unpinned items are removed from the viewport, and the previous 5 items in the sequential ordering of the items replace the 5 items that were removed from the viewport. If 2 items are pinned, the page size is reduced to 4. Pinning works similarly for collections in the writable collections bin [45].

When pinning an element, the pinned element is excluded from the set of elements that are sequentially ordered by the annotation server [61], as shown in some other figures.

- 7 047005

This prevents the pinned item from appearing twice in the viewport. If 2 elements are attached, then these 2 attached elements are excluded from the set of elements sequentially ordered by the annotation server [61]. Excluding elements from sequential ordering is done in addition to resizing the page. Eliminating sequential ordering by the annotation server [61] works in a similar way for collections in the writable collections bucket [45].

Filters, such as the search filter [46] shown in the exemplary embodiment of FIG. 10 are used to receive input from users. In the case of the search filter [46] for the item basket [44], this is the search phrase. These inputs are converted into parameter values sent to the annotation server [61], which is shown in some other figures. These parameter values are used by the annotation server [61] to input parameters into the query that is used by the electronic database [62] shown in some of the other figures. The request includes parameter values and page number. A query with parameter values is used to specify a set of elements and the sequential ordering of those elements. The page number indicates a specific page. Additionally, the request may include a specified number of items that must be sent to the web browser running on the client electronic device. This query contains parameters for filter values such as search phrases, and an item exception list to exclude from a specified list of items, where the query, along with the parameters and the item exclusion list, introduces a sequential ordering of that list of items.

Filters, such as the collection search filter [47] shown in the exemplary embodiment of FIG. 10 are used to receive input from users. In the case of the collection search filter [45] for the writable cart [47], this is the collection search phrase. These inputs are converted into parameter values sent to the annotation server [61], which is shown in some other figures. These parameter values are used by the annotation server [61] to input parameters into the collection query that is used by the electronic database [62] shown in some of the other figures. A collection query request includes parameter values and a page number. A collection query request with collection parameter values is used to specify a set of collections and to order those collections sequentially. The page number indicates a specific page. Additionally, the collection request request may include a specified number of collections that must be sent to the web browser running on the client electronic device. This collection query request contains parameters for the collection filter values, such as the collection search phrase, and a list of collection exceptions to exclude from the specified list of collections, where this collection request request, along with the parameters and the collection exception list, introduces a sequential ordering of this collection list.

After the user creates an annotation, it appears in the item basket [44] in the exemplary embodiment shown in FIG. 10. Using a drag-and-drop process, the user can move or duplicate the annotation element field [36] into the collection, as in the exemplary embodiment depicted in FIG. 11. In the exemplary embodiment shown in FIG. 11 shows a movable block of annotation elements [51]. As the floatable block of annotation elements [51] approaches the displayed record collection [50], as in the exemplary embodiment shown in FIG. 10, the visual form of the displayed recording collection [50] changes as shown in FIG. 11, displaying screen elements including the move target [52] and the duplication target [53]. In this exemplary embodiment of the invention, if the mouse pointer representing dragging of a moving block of annotation elements [51] is located in the area designated by the moving target [52] and is released, the moving action is triggered. In this exemplary embodiment of the invention, if the mouse pointer depicting dragging the field of the moving annotation element [51] is in the area designated by the move target [52] and is released, the move action is triggered in the area designated by the duplication target [53] and is released , the duplication action is started.

In the exemplary embodiment shown in FIG. 12, the user is presented with a closed question annotation character [35], a closed exclamation annotation character [54], a closed check annotation character [55], and a closed incorrect annotation character [56]. These four annotation symbols are examples of the annotation symbols that are displayed. In an exemplary embodiment, the annotation symbols displayed have at least two possible states. These states include open and closed. When closed, the displayed annotation symbol is visible, and the displayed annotation symbols are designed to have low visibility so as not to distract the user while reading the web document. When open, additional information is displayed in the viewing area of the web browser. In the exemplary embodiment of FIG. Figure 7 shows the displayed annotation symbol in the open state. This annotation display symbol represents the question annotation display symbol. In the exemplary embodiment of FIG. 7, the displayed annotation symbol [29] with an open question is shown along with the highlighted annotation text [27] and comment field [31].

- 8 047005

Annotation symbols are associated with elements of the annotation semantic relationship group. Each annotation semantic relationship shows a relationship, typically based on user selection, between the highlighted text [27] and a comment [32], as shown in the exemplary embodiment of FIG. 7. Each relationship in the group of semantic annotation relationships has its own annotation symbol associated with it. For example, a closed question annotation symbol [35], as shown in the exemplary embodiment of FIG. 8 indicates that the annotation creator has marked the annotation comment as a question about the selected annotation text. This closed question annotation symbol [35] allows a user familiar with how the example version works to know that the user can click on the closed question annotation symbol [35] and see a comment on the annotation, which is a question about the selected annotation text [ 27], as shown in Fig. 7, before the user performs the click action.

Upon clicking the annotation symbol [35] with a closed question, as in the example state shown in FIG. 8 becomes an open question annotation symbol [29] showing highlighted annotation text [27] and an annotation comment field [31] as in the exemplary embodiment shown in FIG. 7. In the exemplary embodiment shown in FIG. 8, it is also possible for the highlighted annotation text [27], the open question annotation symbol [29], and the annotation comment field [31] to become visible when the mouse cursor hovers over the closed question annotation symbol [35], as in the exemplary embodiment shown in fig. 8, and secured when pressed. In order to make this process reflective for the user, the highlighted annotation text [27] in FIG. 7 can be assigned a color based on its annotation symbol. In other words, the color of the highlighted text [27] is based on the semantic relationship between the highlighted text [27] and the annotation comment.

To further illustrate an exemplary embodiment, note the following. A closed exclamation point [54] indicates that the annotation creator has marked the annotation comment as an exclamation statement, which indicates that there is something interesting or instructive in the highlighted annotation text. The closed check symbol [55] indicates that the annotation creator has marked the annotation comment as a statement consistent with the highlighted annotation text. And the closed incorrect annotation symbol [56] indicates that the annotation creator has flagged the annotation comment as a statement that disagrees with the highlighted annotation text.

In addition to the annotation symbols that have been discussed and the displayed annotation symbols shown in FIG. 12, an advertisement annotation symbol may be used to indicate another relationship in a group of annotation semantic relationships. This relationship indicates that the annotation comment [32], as in the example state shown in FIG. 7 is an advertisement associated with the highlighted annotation text [27]. In this situation, the annotation symbol [35] representing a closed question, as in the exemplary state of FIG. 8 will be replaced by an annotation symbol displaying an advertisement.

In the exemplary embodiment depicted in FIG. 12, we see several annotation symbols displayed. In this exemplary embodiment, the corresponding question annotation symbol, disagreement annotation symbol, agreement annotation symbol, exclamation annotation symbol, and advertisement annotation symbol represent five relations in the group of semantic annotation relations. These annotation semantic relations are question relations, disagree relations, agreement relations, exclamation relations and advertisement relations.

In one exemplary implementation, machine learning and deep learning models may be used to automatically determine the semantic relationship between the selected annotation text and the annotation comment. Moreover, user data that is collected through users defining annotation semantic relationships and writing comments on the annotation can be used to train deep learning models, and the by-products of these trained models can be used for other natural language processing purposes. Users may be given the option to opt out of using user-generated data to train deep learning-based natural language processing models. A bidirectional long short-term memory (BiLSTM) model with a finite max-pooling layer can be used to compute sentence embeddings. Other machine learning and deep learning techniques can also be used to generate sentence embeddings. Sentence embeddings can be used to determine the similarity of two sentences. After calculating sentence embeddings for all sentences in a collection by analyzing the selected annotation text, annotation comments, and other sentences appearing in items and answers, these sentence embedding vectors can be used to determine the similarity of an element to other elements in the collection or to determine the similarity of two collections. Based on this similarity analysis, items can be suggested to users for reading, engaging in discussion with other users, or adding to a collection. Similar collections can be suggested to users based on collections that the user has searched, read, or contributed to. Additionally, annotations on a web page can link to similar annotations or content on other web pages or documents based on similarity analysis based on sentence embedding vectors. Model

- 9 047005 Convolutional Neural Network (CNN) can be used in combination with a sentence similarity classifier to check the semantic relationships of annotations specified by users. Annotation semantic relationship data provided by users can be used in conjunction with user reputation to validate user selection of annotation semantic relationships and to potentially identify inappropriate content created and posted by users. Bidirectional Encoding Representations from Transformers (BERT) can be used to identify named entities in web pages and web documents and auto-annotate elements of potential interest to the user. These named entities could be objects found on Wikipedia or products offered for sale on online marketplaces. To moderate user annotations, you can use pre-trained BERTs configured with sentiment analysis to flag inappropriate user content.

In the exemplary embodiment depicted in FIG. 13, the user turns on the symbol beacon by pressing the symbol beacon enable button [57] to make the obscured displayed annotation symbols more visible. The character beacon can be in a non-running or running state. When the Character Beacon On button [57] is pressed, the Character Beacon switches from a non-triggered state to a triggered state. The trigger mechanism for the character beacon may also be a sequence of keyboard keystrokes or mouse buttons. The trigger mechanism can also be located in a menu option in the web browser with the ability to annotate [24]. Whether it is a button available in the display area of a web browser [24] with annotation capability, a menu item in a web browser [24] with annotation capability, a combination of keyboard keystrokes and mouse buttons, an input gesture on a touch screen, or an input gesture through an input device ; it is the web browser extension instruction set [25] that makes these triggers available and fires one of them. A beacon [58] for each annotation symbol displayed.

The following are sequence diagrams that show how messages are passed between different entities. Messages include user input into an annotated web browser [24] or a regular web browser, as well as electronic messages between electronic devices and computer systems. A web browser [24] with annotation capability is shown in an exemplary embodiment in FIG. 3. These sequence diagrams apply to a class of example implementations. After presenting the sequence diagrams, variations of some messages will be presented. With a more precise definition of the messages, it will be possible to discuss a specific example implementation.

In fig. 14 is a sequence diagram that is representative of a class of exemplary implementations. In fig. 14, a sequence diagram shows the interaction between a user [59], a web browser [24] with annotation capability, a URL server [60], an annotation server [61], an electronic database [62], and a content delivery network [63]. After a user [59] visits a URL [64], a web browser [24] with annotation capabilities requests a web document [65], the server for the URL [60] returns a web document [66], and then the web browser [ 24] with the possibility of annotation shows the document [67] to the user [59]. Next, the user [59] enters data to create an annotation [68], after which the annotation-capable web browser [24] sends a set of data elements representing the annotation [69] to the annotation server [61], which sends the images for storage if they are [70], to a content delivery network [63], which returns a URL for submitted images [71]. Next, the annotation server [61] sends a command to store the non-graphic set of data elements and URLs from the CDN as an annotation object [72] to an electronic database [62], which confirms the storage of the annotation object [73]. The annotation server [61] then validates the received set of data elements and sends the annotation display set [74] to the annotation-enabled web browser [24], which shows the user [59] a mapped annotation set pointing to the successfully generated annotation [75]. [69] mentions a set of data elements representing an annotation. The set of data elements may vary depending on the implementation details of the example embodiment and the type of annotation.

In fig. 14 is a flow diagram applicable to a class of example implementations. Depending on the characteristics of the bundle of data elements representing an annotation or selected text, an annotation server [61] may record different data, and a web browser [24] with annotation capability may display different views due to the fact that different data is received and different data are available. data to select when generating a display representation of an annotation or selected text. Next, the exemplary embodiment will discuss screenshot bundles and annotation symbol bundles, which may play the role of the data element bundle in FIG. 14. In addition, FIG. 14 may refer to an exemplary implementation in which the annotation display set is a screenshot display set or an annotation symbol display set. The corresponding screenshot set is the screenshot display set, and the corresponding annotation symbol set is the annotation symbol display set. These corresponding pairs are used where in FIG. 14, a data element set and an annotation display set are used.

- 10 047005

A package of screenshots in one exemplary embodiment includes a uniform resource locator (URL) for a document, data to represent a selected portion of text from the document, wherein the selected portion of text is an annotation, and a labeled screenshot of a portion of the document showing the selected portion of text, wherein the selected portion of text is an annotation. visually marked. The screenshot package may also include information to help you find the selected piece of text in the document. This information for finding a selected piece of text in a document typically includes Document Object Model (DOM) information. An annotated screenshot set is a set of screenshots that also includes an annotated commentary [32], as shown in the exemplary embodiment depicted in FIG. 7. Screenshot packages may also include data to represent the annotation symbol. Screenshot packages can be used without annotation [32]. In such cases, data representing the selected text without annotation [32] along with a corresponding tagged screenshot representing the selected text is found in the screenshot display stack.

An annotation symbol package in an exemplary embodiment includes a uniform resource locator (URL) for a document, data to represent a selection of text from the document, where the selection of text is an annotation, a comment on the annotation [32], and an annotation symbol. The exemplary embodiment of FIG. 7 might be a useful reference. The screenshot package also includes information for finding a selected piece of text in a document. This information for finding a selected piece of text in a document typically includes Document Object Model (DOM) information. The annotation symbol packages may also include a labeled screenshot of a portion of the document showing a selected portion of text, wherein the selected portion of text is visually labeled. Depending on the desired behavior, a labeled screenshot may or may not be necessary.

In one exemplary implementation, DOM information for finding a selection of text in a document may include information for finding an element in the DOM that contains the first character of the selected text, and for finding an element in the DOM that contains the last character of the selected text, as well as the selected text itself. . These two elements are called the start selection element and the end selection element. The information recorded about the start element of the selected text and the end element of the selected text are tag, id, class, index, content, xpath and parent. In addition, the ancestor element of the starting element of the selected text is recorded. For the ancestor element of the starting element of the selected text, the tag, id, class, index, and content are recorded. To find the specified selected text at a later point in time, the DOM is searched by tag and id, then by xpath, then by tag, class, and index, then by tag and class, then by tag and index, and then by tag alone. A simple xpath search is not sufficient because edits may have been made to the web page, or the DOM structure of the web page may have changed as a result of the execution of JavaScript code that inserts elements into the DOM. In this example, the goal is to find the selected text even if edits have been made to the web page. To make it easier to find selected text in different scenarios, several search methods are used. Once the start element of the selected text and the end element of the selected text are found, finding the selected text becomes easier.

Another problem is the insertion of elements into the DOM of a web page by JavaScript code on a visited web page. These inserted elements can also be annotated. The browser's mutation observer can be used to facilitate this process. Once changes are detected, you can rerun the DOM structure detection code. If the start element of the selected text and the end element of the selected text are found, the selected text can be pasted. For selected annotation text, you can also insert a display annotation symbol.

If the passed screenshot set or annotation symbol set contains a tagged screenshot, a storage set of images is created from that tagged screenshot through various operations including resizing, cropping, and modifying the image. These operations are performed to create images that look good in different viewing areas on different screen sizes. Once you've created a set of tagged screenshots for storage, it's worth determining whether those tagged screenshots should be stored on a content delivery network (CDN). For miniature labeled screenshots, it may be advantageous to store the data along with all other data in a screenshot stack or annotation symbol stack. For large marked-up screenshots, it is usually better to store these images using a CDN, which can also make these images available for delivery. An annotation server [61], as shown in some figures, typically decides which images from a storage set should be stored in an electronic database [62], as shown in some figures, and which to send to a CDN. The CDN usually responds with a URL when the image is saved. These URLs are stored along with the rest of the screenshot package or annotation symbol package.

The delivered screenshot is either the actual image or a URL stored in an electronic database [62], as shown in some figures. When data is transferred to the client's mouth

- 11 047005 swarm, URL can be used to get image from CDN. If the image is transmitted directly, it can simply be displayed. Ultimately, it is the annotation object that is stored in the electronic database [62]. An annotation object typically includes all received data in a screenshot burst or annotation symbol burst, along with transmitted screenshots for each tagged screenshot in the storage set.

To enable advertisers to deliver relevant advertising to customers related to the content that users are currently engaging with, ad screenshot sets and advertising annotation character sets could potentially be deployed along with corresponding ad screenshot display bundles and advertising annotation symbol display bundles. The advertisement screenshot display set is a visually displayable form of data in the advertisement screenshot display bundle, and the advertisement annotation symbol display set is a visually displayable data form in the advertisement annotation symbol display bundle. fig. 7 may be used for illustrative purposes, but the displayed annotation symbol is replaced by the displayed advertisement annotation symbol. In an exemplary embodiment of the invention, the set of display advertising symbols contains an annotation comment [32] created by the advertiser that relates to the selected annotation text [27]. The advertisement annotation symbol set contains an annotation symbol that represents an advertisement. Although annotation character sets contain an annotation character, a screenshot set can also include an annotation character. While screenshot sets contain a tagged screenshot, an annotation symbol set can also contain a tagged screenshot. fig. 9 can be used for illustration, but the annotation symbol displayed is replaced by the advertisement annotation symbol displayed. Additionally, the annotation comment is created by the advertiser along with other elements, but the overall form should resemble other blocks of annotation elements. When tagged screenshots are included in a system, developing a productive system may require creating storage sets and using a CDN to distribute those tagged screenshots. In an exemplary implementation, these advertising screenshot sets and advertising annotation symbol sets are often stored and managed through an advertising server that communicates with the annotation server. The ad server is not shown explicitly in sequence diagrams, but can be inserted. To include one advertisement, the page size can be reduced by one. After the annotation server [61], as shown in some figures, prepares a page of elements representing annotations, the annotation server [61] can contact the declaration server to obtain 1 declaration.

The semantic web provides semantic descriptions of some web pages and documents available on the WWW using ontologies, which are implementations of a certain type of description logic. These ontologies must be created for different domains, and these ontologies have the ability to express concepts within that domain in a logical form, as is done in description logic. Collections of already existing ontologies are available through projects such as KBPedia and others. Description logic is a family of logics defined in mathematical logic that have some of the characteristics of first-order logic. Depending on the properties of the logic family, they have different computational characteristics. Using an existing ontology, it is usually possible to make logical statements. In one exemplary embodiment, these logical statements can be translated into natural language, and some natural language sentences can be translated into a statement made within description logic using neural machine translation techniques. Other deep learning methods can also be used. Machine learning and deep learning models can translate some natural language sentences into statements in a specific ontology. In this exemplary embodiment, this methodology can be used for advertising purposes. An advertiser can express concepts and write logical statements about a product or service within an ontology related to the product or service. These logical statements can then be entered into the advertising platform. The ad server can identify web pages and documents containing natural language text that can be translated into a statement in an ontology selected by the advertiser to express a concept about the advertised product or service. In addition, the ad server can determine which of these web pages and documents contain natural language text that expresses statements similar to or logically related to the statement expressed by the advertiser. After identifying these web pages and documents, as well as the corresponding location in the document, the advertiser may be given the opportunity to place an advertisement at that specific location on the web page or document, or somewhere that has a link to that specific location. Using logic analyzers, you can determine the various consequences of statements that appear on web pages or in documents. Using this information, you can narrow down the set of web pages or documents and determine their location in the document to find the ones that are most relevant to the advertiser. With the help of subtitles, voice recognition and text summarization, images, videos and audio recordings can be translated into text. In some cases, this text can be translated into statements within the same ontology used by the advertiser. Logic analyzers can then be used to determine

- 12 047005 the most suitable candidates for display to the advertiser for consideration by the advertiser. Based on the advertiser's selection, the advertising platform can generate ads to be placed on the ad server. Moreover, in this exemplary embodiment, ontologies can be created for the purpose of expressing facts about products and services for sale in various fields, such as insurance, financial services, legal services, and healthcare. These ontologies can then be used by various groups advertising and selling services in these areas to create logically correct and expressive statements about their products and services for subsequent searches of media such as web pages, web documents, images, videos and audio recordings. These media can then reach the audience consuming the media, and the advertiser can know that this audience is likely to be familiar with the claims made by the advertiser in the ontology. Moreover, in this exemplary embodiment, the media source itself has a logical relationship with the advertiser's claim, and the advertiser can use this logical relationship to provide more specialized products and more specialized targeting to meet the customer's needs.

In fig. 14, to clarify the details of the exemplary embodiment, it is necessary to specify a set of data elements along with details about the storage set and the screenshot being delivered. A set of data items may be a collection of screenshots with corresponding data items. The set of data elements may also be a set of annotation symbols with corresponding data elements. You should determine the storage set, size, resolution, cropping, and other properties of the different types of tagged screenshots. This is usually based on considerations about the type of display these labeled screenshots will be displayed on, as well as the size of the view being displayed. In some cases, the storage set will contain thumbnails of the tagged screenshots. If these thumbnails are small in size, storing these images using a CDN may not be the most efficient approach. It may be more efficient to store the image in an electronic database [62], a local file system, or other local location, as opposed to a content delivery network [63]. Specifying whether the screenshot being delivered is a real image or a URL stored in an electronic database is another decision that should be taken into account to clarify implementation details. Those skilled in the art will appreciate that various minor variations may be introduced for storing and retrieving images. It will also be understood by those skilled in the art that these variations are interchangeable and the differences are not significant.

The electronic database [62] shown in FIG. 14 and some other figures, can take various forms, from a relational database to a non-relational database. Additionally, an electronic database [62] may be a distributed electronic database system implemented on one or more computer systems. The most beneficial database features may vary depending on the number of database clients and the usage patterns of those database clients. In the case of an exemplary implementation, the annotation server [61] is a database client. To handle large numbers of annotation servers with varying consistency, availability, and persistence needs, as well as varying latency requirements, the characteristics of an electronic database or distributed electronic database system may vary. In addition, distributed electronic database systems can be implemented in a variety of ways.

Additionally, it should be noted that storing images in an electronic database [62] can sometimes impact performance. In such cases, it is beneficial to store references in a database and use a different storage mechanism for certain images. In the case of a content delivery network [63], the link is a URL. However, it is possible to store a link that is not a URL and then use an alternative mechanism to store the image so that it can be retrieved using the link stored in the electronic database. In an exemplary embodiment, the image may be stored on a local file system, and the link in the database may correspond to the path to the file. In such cases, the electronic database and storage location should be considered as a distributed electronic database storing the image in a more efficient manner. Those skilled in the art will appreciate that many minor variations of this method can be used to store and retrieve images. It will also be understood by those skilled in the art that these variations are interchangeable and the differences are not significant.

In fig. 15 is a sequence diagram that is a class of exemplary implementations. In order for fig. 15 defined details of an exemplary implementation, it is necessary to specify annotation display sets and annotation display sets. In addition, FIG. 15 may apply to an example embodiment in which annotation display sets are replaced by screenshot display sets, and annotation display sets are replaced by screenshot display bundles. In this case, you must specify the screenshot display sets with the data elements they should include, as well as specify the mechanism for receiving the delivered screenshots. UI diagrams, data elements described for sets of screenshots, and reviewed

- 13 047005 earlier considerations can be used to define annotation display sets and annotation display sets. The displayed screenshot package is typically used when there is an annotation highlight and a tagged screenshot with no annotation comment. The Annotation Screenshot Display Pack is used when you have a tagged screenshot, annotation highlight, and an annotation comment. In fig. 15 sequence diagram shows the interaction between a user [59], a web browser [24] with annotation capability, an annotation server [61], an electronic database [62], and a content delivery network [63]. After the user [59] has visited the URL of the annotation server [76], the annotation-capable web browser [24] requests the organization page [77] of the annotation server [61], which requests the annotation objects required for the organization page [78] ,from an electronic database [62] that returns annotation ,objects [79] to the annotation server [61]. The annotation server [61] then returns the annotation server organization page with the annotation display sets [80] to the annotation-capable web browser [24], which requests images for the annotation display set URLs returned when returning the annotation server organization page [81] . Finally, the content delivery network [63] returns the requested images [82] to the annotable web browser [24], which shows the user [59] an organize page with the annotation sets displayed [83]. Next, the user [59] enters input to move the annotation [84] into a web browser [24], which requests that the annotation [85] be moved by the annotation server [61], which sends instructions to move the annotation object [86] to the electronic database [62 ]. In response, the electronic database [62] confirms the movement of the annotation object [87] to the annotation server [61], which confirms the movement of the annotation [88] to the annotation-capable web browser [24], which shows the user [59] the content with the displayed bundle of annotations in new place [89].

Annotation screenshot display sets or annotation object blocks are created using the data in annotation screenshot display sets sent from the annotation server [61], as shown in some of the drawings. An annotation screenshot display set is created using data in annotation objects extracted from an electronic database [62], as shown in some of the drawings. When viewing annotation pages in a screenshot collection view area, such as the item basket [44] of the annotation organization contextual area [43] of an organization's web page, as shown in the exemplary embodiment of FIG. 10, the data sent by the annotation server [61] is a page of thumbnail display sets. These thumbnail display sets include at least a deliverable screenshot for a tagged screenshot in an annotation object storage group stored in an electronic database [62]. fig. 9 may be used for illustrative purposes. In an exemplary embodiment of the invention, the thumbnail display sets also contain data in a human-readable form that can be displayed by a web browser, representing a single highlighted contextual annotation text [38], a single contextual annotation comment [40], and a single contextual question annotation display symbol [39]. . In addition, the highlighted text of an individual contextual annotation [38] and the displayed symbol of an individual contextual question [39] are color-coded. All these pieces of data working together give the user more information at a glance. A displayed bunch of annotation screenshots can also be shown on the annotation details web page to provide more detailed information about a single annotation along with comments and responses from many users about that annotation.

In fig. 16 is a sequence diagram that is a class of exemplary embodiments. In order for fig. 16 explicitly indicated an exemplary implementation, it is necessary to specify annotation symbology sets and corresponding annotation symbology displays with data elements to be included along with the visual appearance. User interface diagrams, the data elements described for annotation character sets, and the considerations discussed earlier can be used to achieve this refinement. In fig. 16 shows the interaction between a user [59], a web browser [24] with annotation capability, a URL server [60], an annotation server [61], and an electronic database [62]. After a user [59] has visited a URL [90], the annotation-capable web browser [24] requests the web document [91] from the URL server [60], which returns the web document [92] to the annotation-capable web browser [24]. annotation. Next, the annotation-capable web browser [24] requests the annotation symbol display sets representing the annotations [93] from the annotation server [61], which queries the annotation objects [94] from the electronic database [62]. The electronic database [62] then returns the annotation objects [95] to the annotation server [61], which returns the annotation symbol display sets representing the annotations [96], to the annotation-enabled web browser [24], which displays the web document with closed display annotation symbols and annotations that can be opened by hovering or clicking [97].

In fig. 17 is a block diagram that is representative of a class of exemplary embodiments. A specific exemplary embodiment can be determined if the set of data elements is precisely specified,

- 14 047005 which are accepted and stored. For example, we can explicitly specify a set of screenshots with all of its data elements, or we can explicitly specify a set of annotation symbols with all of its elements. In this case, the set of screenshots or the set of annotation characters is the set of data elements that are accepted. In addition, you must specify the set and storage mechanism for the transmitted screenshots if the set of data elements contains a tagged screenshot. These details help to illustrate the embodiment. In one exemplary embodiment, the storage set may include a labeled screenshot for display on large monitors, laptops, tablets, and phones, as well as a miniature labeled screenshot for item boxes. The exact dimensions may be adjusted to suit the most common sizes of these displays currently in use. Additionally, a server in a client-server architecture will receive a batch of data elements representing the annotation [98] and store the received data elements and any newly created elements using one or more computer systems [99] to satisfy requests made by the web browser the user's client device creating the annotation or simple highlight. In fig. 17 is a flow diagram of a method implemented by one or more computer systems representing a server in a client-server architecture. This method is implemented by the server to enable sharing of data that may be requested by the client user interface of the same client or another client. The steps of this method are to obtain a package of data elements representing the annotation [98] and store the resulting data elements and any newly created elements using one or more computer systems [99].

In fig. 18 is a block diagram that is representative of a class of exemplary embodiments. A specific exemplary embodiment may be specified if the data elements included in the set mapping set are precisely defined. This means specifying the data elements in the display set. In one example embodiment, the display sets may be screenshot display sets or annotation symbol display sets. If the display sets are screenshot display sets, then the display set will be the screenshot set display set in FIG. 18. If the display sets are annotation symbol display sets, then the display set will be the annotation symbol set display set in FIG. 18. In addition, a server in a client-server architecture will compile a display set, including a page of display sets, using one or more computer systems [100] and transmit the display set in a format that allows the display sets to be displayed by a web browser running on electronic device [101] to satisfy requests made by an annotation-capable web browser running on the client device of a user viewing an annotation page in a web document or individual context area. The block diagram shown in FIG. 18 is implemented by one or more computer systems.

In fig. 19 is a block diagram of a class of exemplary embodiments of the invention.

A specific exemplary embodiment may be indicated after explicitly specifying a set of data elements. For example, you can explicitly specify a pairing of screenshots with data elements to include, or you can explicitly specify a pairing of annotation symbols with data elements to include. In fig. 19 is an annotation creation flowchart showing the steps performed by a set of web browser extension commands that in an exemplary embodiment converts a web browser that supports adding browser extensions into an annotation-capable web browser [24], as shown in some of the drawings. . In fig. Figure 19 shows the steps taken by the user [59], web browser [102], web browser extension content script [103], and web browser extension background script [104] when creating an annotation. The user [59] opens the browser [105], then the web browser [102] downloads the browser extension [106]. The user [59] then visits the URL [107], after which the web browser [102] downloads the data sent from the visited URL [108] and injects the content script [109]. Now user [59] is browsing web page [110] and highlighting text [111]. At this point, the web browser extension content script [103] makes the annotation symbol selection button bar [112] available as a result of executing the web browser extension's instructions. Next, the user [59] clicks the button corresponding to one annotation symbol in the annotation symbol selection button bar [113], and the web browser extension content script [103] responds by making the annotation input field [114] available. Next, the user [59] enters text and presses the Enter key [115], and the web browser extension content script [103] sends a batch of data items to the background script [116]. At this point, the web browser extension background script [104] receives the data item package from the content script [117] and sends the data item package to the annotation server [118].

In fig. 19, after the user [59] enters text and presses the Enter button [115], the web browser extension content script [103] should generate either an annotation symbol package or a screenshot package. In accordance with an exemplary embodiment of the invention, the annotation symbol package would include a tagged screenshot. If the labeled screenshot is gene

- 15 047005 is server-side rendered, the web page may have to be re-rendered or labeled as a screenshot. This is computationally expensive and prone to errors. Generating a labeled screenshot using a browser extension does not encounter these issues. The browser extension must protect the user's privacy. Screenshots must not include the desktop or other browser tabs. Additionally, in an exemplary implementation, the screenshot would be marked with highlighted annotation text [27] and an annotation comment field [31]. The highlighted text [27] and comment field [31] are shown in FIG. 7, which may be useful for illustration in this case. In an exemplary embodiment of the invention, a simple delay may be implemented before the screenshot is taken. This delay should be sufficient for the highlighted annotation text [27] and annotation comment field [31] to be present on most devices. An alternative exemplary embodiment could implement a more performant method in which a web browser extension content script [103] looks for the presence of annotation highlight text [27] and annotation comment field [31] in the document's DOM structure and takes a screenshot if these elements are present . Searching for the presence of highlighted text [7] and annotation comment fields [31] in the DOM structure of documents will occur at periodic time intervals that are shorter than the simple delay time discussed earlier. In an exemplary embodiment, the periodic interval may be one third of the length of the simple delay that would be selected.

In fig. 20 is a block diagram of a class of exemplary embodiments. A specific exemplary embodiment may be specified after the displayable annotation character set has been explicitly specified with all of its data elements. In fig. 20 is an annotation viewer flowchart illustrating the steps performed by a web browser extension instruction set that, in an exemplary embodiment, converts a web browser that supports browser extensions into an annotation viewer-capable web browser. In fig. 20 shows the actions performed by the user [59], web browser [102], web browser extension content script [103], and web browser extension background script [104] when viewing an annotation. The user [59] opens the browser [119] and the web browser [102] downloads the browser extension [120]. The user [59] then visits the URL [121], after which the web browser [102] downloads the data sent from the visited URL [122] and injects the content script [123]. The web browser extension background script [104] now receives the annotation character display sets representing the annotations for the URL [124] and sends the annotation character display sets to the content script [125]. The web browser extension content script [103] now receives annotation symbol display sets from the background script [126] and displays the annotation symbols in a closed state on the web page [127]. Now the user [59] is viewing a web page with the annotation symbol display closed [128] and clicks on the annotation symbol display symbol [129] that is closed. In response, the web browser extension content script [103] displays the annotation symbol in clear text [130]. Next, the user [59] views the web page with the selected displayed annotation symbol in clear view [131].

In the exemplary embodiment depicted in FIG. 31, a web page of collection items is shown. The Collection Items Page context area [258] represents the display area of the items displayed, the collection item paging mechanism [259], the collection item search filter [260], and the collection action menu trigger [261]. The Collection Actions menu [262] displays the option to send a message to the collection [263], the option to invite a user [264], and the option to change user permissions [265]. The Collection Action Menu Trigger [261] is used to launch the Collection Action Menu [262]. The collection item paging mechanism [259] can be used to change the display collections page displayed in the context area of the collection item page [258]. The Send Message to Collection option [263] creates a message thread with the first text message and sends it to collection members who have read permission or higher for the collection. The message thread appears in the user's inbox. An exemplary embodiment of a user's inbox is shown in FIG. 33. In the exemplary embodiment of FIG. Figure 33 shows a message thread from the US History collection [281]. A message sent to members of this collection using the Send Message to Collection option [263] will appear in the recipients' message inboxes assigned to the collection to which the message was sent. All collection members with Read permission or higher will be able to read the message, and all collection members with Send Message permission or higher will be able to reply to it. The Invite User option [264] can be used to invite new users to the collection. Invitations are sent to the invited user's inbox. The Change User Permissions option [265] can be used to change the permissions of a member of this collection. Notifications in the form of a change message thread can be sent to the inbox of the user whose permissions have been changed.

Collection element filters, such as the collection element search filter [260] shown in the exemplary embodiment of FIG. 31, are used to receive input data from the user

- 16 047005 bodies. In the case of the Collection Items Page context area [258] of the Collection Items Search Filter [260], this is the collection items search phrase. These inputs are converted into collection element parameter values that are sent to the annotation server [61], as shown in some of the figures. These collection item parameter values are used by the annotation server [61] to input the collection item parameters into the collection item query query that is used by the electronic database [62], as shown in some of the figures. The collection item request request includes the collection item parameter values and the page number of the collection item. A collection element query request with collection element parameter values is used to specify a set of collection elements and the sequential ordering of these collection elements. The collection item page number identifies the specific page of the collection item. Additionally, a collection item request may include a specified number of collection items that must be sent to a web browser running on the client electronic device. This collection item request contains collection item parameters for collection item filter values, such as a collection item search phrase and a list of collection items to exclude from a specified list of collection items and sequential ordering of that list of collection items.

In the exemplary embodiment depicted in FIG. 32, the collections web page is shown. The Collections Page context area [266] represents the display area of the collections, the collections page paging mechanism [267], the collections page search filter [268], and the collections page action menu trigger [269]. The Actions menu on the Collections page [270] displays options for Ownership Filter [271], Administration Filter [272], Write Filter [273], Message Filter [274], Read Filter [275], and Subscribe Filter [276]. The Collections Page Action Menu Trigger [269] is used to launch the Collections Page Action Menu [270]. The collections page paging mechanism [267] can be used to change the display collections web page displayed in the collections page context area [266]. The Ownership Filter Option [271], the Administration Filter Option [272], the Write Filter Option [273], the Message Filter Option [274], the Read Filter Option [275], and the Subscription Filter Option [276] filter collections on based on user permissions. Ownership is the highest level of permission, and ownership is granted to the creator and owner of the collection. Ownership allows a user to grant other users administrative rights to that collection and do anything allowed by lower-level permissions. Administrator permission is lower than Owner permission and allows users to elevate non-administrator permissions and perform other administrative tasks, as well as anything allowed by lower-level permissions. The Write right is lower than the Administrator right and allows users to add items to the collection and do anything that is allowed by lower level rights. The send message right is lower than the write right and allows the user to send messages to other members of the collection and do everything that the read right allows. The Read right is lower than the Message right and allows the user to view all items in a collection and read all messages in a collection for that collection. The subscription right gives the user the right to read the public collection. Subscription permission is typically granted without administrative intervention to the user who requests the permission because the collection and its associated messages are public. To allow users to visualize their privacy settings, background colors are assigned based on colors. Colors associated with shorter wavelengths imply greater resolution, and colors associated with longer wavelengths imply less resolution. In other words, the colors of the rainbow are related to resolutions. The color purple represents owned collections. Blue color corresponds to collections available for administration. Green color corresponds to writable collections. Yellow corresponds to collections available for messages. Orange corresponds to collections that are available for reading. And the red color matches the subscription collections. For design purposes, you can use pastel versions of these colors. Or you can use a different set of backgrounds to represent a spectrum of values. These backgrounds are applied to the displayed collections in the exemplary embodiment of FIG. 32. In the exemplary embodiment of FIG. 31 web page backgrounds change to match the color associated with the permissions in this collection.

In one exemplary implementation, to facilitate viewing of large numbers of items, including annotations, that are found in a single collection, annotations may be grouped into stacked sets of displayable annotation items. These stacked sets of annotation display elements appear as a single element, but visually they appear as stacked elements. It is advantageous to place annotations from the same url, up to 9 annotations or other fixed custom value, annotation display elements, in a stacked set of annotation display elements. To facilitate this mapping, the paging mechanism must support returning elements in this format. To illustrate, imagine that 8 out of 10 elements are annotation elements belonging to the same url. When these 8 elements are grouped into a stacked set of annotation display elements, the element page

- 17 047005 collection 3 elements remain. To ensure that 10 elements are present even when using stacked sets of annotation display elements, the annotation server must return more elements [61]. An example embodiment uses a relational database [246] and may require a complex structured query in query language (SQL) using window functions. Typically, to access a relational database, most modern web frameworks use object-relational mapping (ORM) systems [246]. In the exemplary implementation, the ORM system did not handle window functions. To overcome this limitation, a relational algebra library was used, which is used by the ORM system. Using this relational algebra, a complex SQL query was generated using appropriate window functions. This allows you to display collection item pages with stacked sets of annotation items to display.

Collection filters, such as the collections page search filter [268] shown in the exemplary implementation of FIG. 32 are used to receive input from users. In the case of the collections page context area [266] of the collections page search filter [268], these are collections search phrases. These inputs are converted into collection parameter values sent to the annotation server [61], as shown in some figures. These collection parameter values are used by the annotation server [61] to input the collection parameters into the collection query request that is used by the electronic database [62], as shown in some of the figures. The collection query request includes the collection parameter values and the collection page number. A collection query request with collection parameter values is used to specify a set of collections and to order those collections sequentially. The collection page number identifies a specific collection page. Additionally, the collection request request may include a specified number of collections that must be sent to the web browser running on the client electronic device. This collection query request contains collection parameters for the collection filter values, such as the collection search phrase and the list of collections to exclude from the specified list of collections and the sequential ordering of that list of collections.

In the exemplary embodiment depicted in FIG. 33, the incoming messages page is shown. The Collection Message Thread Section and Header [277] includes the US History Collection Message Thread Section and Header [279] and the Car Collection Message Thread Section and Header [280]. Collection message threads are message threads associated with a collection. These collection message threads are located in the Collection Messages section under the name of the collection to which the collection message thread belongs. In this example embodiment, the section and header [279] of the US History Collection Message Thread indicates the US History Collection message thread [281], and the section and header [280] of the Automobile Collection Message Threads indicates the first Automobile Collection message thread [282] and the second thread car collection messages [283]. Unlabeled message threads are those message threads that are not associated with any collection. Unstarred message threads appear in the Unstarred Messages section. In an exemplary embodiment, the section and header [278] of the Unmarked Message Thread includes unmarked message thread one [284], unmarked message thread two [285], and unmarked message thread three [286]. When a user clicks on a message thread, the message thread appears in the message thread contextual area [287]. The contextual message thread area [287] displays the first system message of the message thread US History [288] with an accept system message button [289] and a reject system message button [290]. The user can press the system message accept button [289] to accept the invitation to join the collection, or the user can press the system message reject button [290] to decline the invitation to join the collection.

In the exemplary embodiment depicted in FIG. 34, the inbox page is displayed after the user accepts the invitation to join the collection. System message [291] indicates that the user has accepted the invitation to join the collection. The message thread separator [292] separates one user's messages from another user's messages. Those skilled in the art will also appreciate that the message thread separator [292] can be implemented using various visual design techniques to separate one user's messages from another user's messages in a given context. The following shows the first text message [293], second text message [294], and third text message [295] of the US History message thread. In addition, the message contents section and header [296] are shown, as well as the message contents window [297] and the send message button [298]. This inbox has a permissions management system and links to collections. When a user gains permission for a sliver of messages, all other messages in that thread become visible, even if the user did not initially have access when creating the message thread. This facilitates discussion, and access is controlled at the message thread level. In addition, system message threads typically ensure that collection administrators are aware of the activities and permissions of group members. Additionally, users with message rights can send

- 18 047005 chain messages with one message to users in a collection or a select group of people in a collection. The ability to create a collection thread helps clarify the subject matter and direct the user's attention to the collections or tasks they want to focus on.

Inbox is an effective means of inviting users to collections if the user is already on the site. Inboxes are also effective for exchanging items with others if the user is already on the site. To facilitate the exchange of items and collections when the user is not on the site, links with cryptographic hashes are used. These links are impossible to guess, and the user having the link does not need to create an account to access the link. To exchange, you just need to have a link. Additionally, public items, including annotations and collections, can be shared via social media.

In fig. Figure 21 shows a flowchart of a nurse using an example web browser [24] with annotation capabilities, as shown in some of the figures, with a notepad function. The nurse wants to teach the client that rice is not good for diabetics and then show the client that a favorite food can be modified to make it more diabetic-friendly. Once the notebook is created, it can be shared with patients. On the first day [132], the nurse visits the glycemic index page on Wikipedia, highlights the American Diabetes Association endorsement, and creates a control annotation along with a comment on the annotation: Most physicians strongly believe that foods with a high glycemic index negatively affect the health of diabetics. [134] and adds the annotation to the notepad [135]. Next, the nurse visits allrecipes.com and finds the dish Flavored Spanish Rice and Beans, identifies 1 cup of uncooked white rice, and creates an exclamation blurb along with a comment on the blurb that reads: Rice has a very high glycemic index, but you can make the same recipe with pearl barley . [136] and adds an annotation to the notepad [137]. The nurse then goes to the Harvard Health website and finds the Glycemic Index page for 100+ foods, highlights Cooked White Rice* with an exclamation point, and says: Most people don't realize that white rice has such a high glycemic index. Note that it is 73, which is higher than most foods. [138] and adds an annotation to the notepad [139]. Next, the nurse goes to the Harvard Health website and finds the Glycemic Index page for 100+ Foods, highlights Pearl Barley with a dagger annotation, and says: Most people don't realize that Pearl Barley has such a low glycemic index. Note that it is 28, which is lower than most foods. [140] and adds the annotation to the notepad [141]. The nurse then goes to Target's website, finds the product page for pearled barley, and selects Pearled Barley - 1lb - Market Pantry with a dagger blurb that says: Pearled barley can be cooked in a rice cooker just like rice. What's more, cooked pearl barley has the look and feel of grain, making it an ideal substitute for rice. They can replace rice in recipes that use rice; can be substituted for rice in recipes that call for rice. [142] and adds the annotation to the notepad [143]. After creating the notebook, the nurse on the second day [133] shares the notebook with a patient who has diabetes and likes to cook Spanish rice [144].

In fig. Figure 22 shows a flowchart of a patient using an example web browser [24] with annotation capabilities and a notepad function to view a notepad created by a nurse. The patient on day two [145] opens the notepad link and clicks on the first displayed annotation screenshot embedded in the notepad [147] and reads the Wikipedia title Glycemic Index and reads the highlighted text American Diabetes Association supports after observing the check mark displayed by the annotation symbol and then reads Commentary on the abstract: Most doctors firmly believe that foods with a high glycemic index negatively affect the health of diabetics [148]. Next, the patient clicks on the second screenshot of the abstract embedded in the notepad [149] and reads the headline allrecipes.com Flavorful Spanish Rice and Beans, then reads the highlighted text 1 cup of uncooked white rice after seeing the exclamation mark in the abstract, and then reads the commentary on annotations: The glycemic index of rice is very high, but you can prepare the same recipe with pearl barley. [150]. The patient then clicks on the third displayed annotation screenshot embedded in the notepad [151] and reads the title of the Harvard Health Glycemic Index of 100+ Foods website, then reads the highlighted text White rice boiled*, seeing the exclamation symbol displayed on the annotation, and then reads the commentary on annotations: Most people don't realize that white rice has such a high glycemic index. Note that it is 73, which is higher than most foods [152]. Next, the patient clicks on the fourth displayed annotation screenshot embedded in the notepad [153] and reads the site title Harvard Health Glycemic Index of 100+ Foods, then reads the highlighted text after seeing the dagger annotation symbol displayed, then reads Pearled Barley, then reads Commentary on the abstract: Most people don't realize that pearl barley has such a low glycemic index. Note that it is 28, which is lower than most foods [154]. After this, the patient clicks on the fifth displayed annotation screenshot embedded in the notepad [155], which

- 19 047005 melts the Target site name and notices that it is a product page for pearled barley, reads the highlighted text Pearled Barley - 1lb - Market Pantry after seeing the dagger annotation symbol displayed, and then reads a comment on the annotation that reads: Pearled Barley Can cook in a rice cooker the same way as rice. What's more, cooked pearl barley has the look and feel of grain, making it an ideal rice substitute. It can be used as a substitute for rice in recipes that call for rice. [156]. Having understood the contents of the notebook, the patient, on the day of grocery shopping [146], goes to Target, buys pearled barley and prepares Flavored Spanish Barley and Beans [157].

In the exemplary embodiment depicted in FIG. 23, a web browser [24] with annotation capabilities and a notepad function is presented. A web browser with annotation capability and notepad functionality monitors the presence or absence of an active notebook and allows the user to select the active notebook from a group of notebooks available to the user. In fig. Figure 23 shows an active notebook [159] and an annotation built into the notebook [160]. When an active notebook [159] is present, the annotation-enabled notepad web browser is in the active notepad state, and when an active notepad [159] is not present, the annotation-enabled notepad web browser is in the active notepad state. The displayed notebooks include many annotations built into the notebook [160]. An embedded annotation [160] is a container that includes a display set of annotations showing the data contained in the annotation display bundle. In an exemplary embodiment, the annotation embedded in the notebook [160] includes a displayable annotation display set that also includes an annotation symbol. In an alternative embodiment, the annotation embedded in the notebook [160] may include a displayed bunch of screenshots without the annotation symbol or annotation comment. The user can add an annotation to the notebook by clicking on the add to notebook button [158] shown in FIG. 23. The notebook includes a list of references to annotation objects. The electronic database [62], as shown in some of the figures, stores notebook objects along with a list of links to annotation objects. The annotation server [61], as shown in some figures, retrieves the notebook object and annotation objects from the electronic database [62] and sends the notebook to the annotation-capable web browser [24] when it receives a request from the annotation-capable web browser. -browser [24]. Once the active notebook [159] is displayed as the display notebook, the user can navigate to the annotation by simply clicking on the notebook's embedded annotation [160], as shown in FIG. 23 as an annotation built into the notebook [160]. In some exemplary embodiments of the invention, each pad is associated with a stack display set. In one exemplary embodiment of the invention, each notebook may be associated with a display set of screenshot stacks. In another exemplary embodiment, each notebook may be associated with a display set of stacks of annotation screenshots.

The notebooks include a wealth of information through links to other sources along with annotated comments. This information could potentially be supplemented by descriptions written by the notebook's creator. Using these notebooks, you can create presentations in a variety of formats, including PDF, Microsoft Word, and other open source and proprietary document formats. These presentations are created by sequentially including, for each annotation, a corresponding screenshot of highlighted text [37], a separate highlighted text of the contextual annotation [38], a separate highlighted annotation symbol [39], and a separate highlighted text of the annotation [40], as shown in FIG. 9. Additionally, this information can be supplemented with text annotations of related articles generated using natural language processing techniques as well as image captioning techniques. Additionally, the user can create a written description that will be added at certain points, and these written descriptions will be present in the presentation output.

In the exemplary embodiment depicted in FIG. 24, a flow diagram shows, in an exemplary embodiment, the interaction between a user [59], a web browser with annotation capability and notepad functionality [161], a URL server [60], an annotation server [61], an electronic database [62] and a content delivery network [63] to create a notebook. First, the user [59] visits the URL [162] in a web browser with the notepad function [161], which requests a web document [163] from the URL server [60], which returns a web document [164] to the web browser with the notepad function notepad [161], which shows a web document [165] to the user [59]. Next, the user [59] enters data to create the notebook Aromatic Spanish Barley and Beans [166] in a web browser with a notepad function [161], which sends a request to create a notebook Aromatic Spanish Barley and Beans [167] to the annotation server [61], which sends instructions for creating a notepad object [168] to an electronic database [62]. The electronic database [62] then confirms the creation of the notebook object [169] to the annotation server [61], which confirms the creation of the notebook [170] to a web browser with notepad functionality [161], which shows the displayed notebook [171] to the user [59]. Next, the user [59] enters data to create a glycemic index annotation [172], which sends a batch of annotation screenshots [173] to an annotation server [61], which optionally sends images for storage [174] to a content delivery network [63]. Content Delivery Network

- 20 047005

[63] returns the URL for the stored images [175] to the annotation server [61], which sends instructions for creating a glycemic index annotation object with a CDN URL [176] to the electronic database [62]. Next, the electronic database [62] acknowledges the creation of the annotation object [177] to the annotation server [61], which acknowledges receipt of the batch of annotation screenshots and sends the batch of annotation screenshots [178] to the notepad-enabled web browser [161], which displays the displayed batch of screenshots annotations [179] to user [59]. Next, the user [59] adds the glycemic index annotation to a notebook [180] in a web browser with the notepad function [161], which requests a link to the glycemic index annotation object to be added to the notebook [181] by the annotation server [61], which sends instructions to adding a link to the glycemic index annotation object in the notebook [182] to the electronic database [62]. The electronic database [62] then confirms the addition of the annotation object link [183] to the annotation server [61], which confirms the addition of the glycemic index annotation object link to the notepad [184] to the web browser with notepad function [161], which displays the web document with a notepad on the side displaying a screenshot of the annotation added by [185] to user [59].

In the exemplary embodiment depicted in FIG. 25, the sequence diagram shows, in an exemplary embodiment, the interaction between a user [59], a web browser with annotation and notepad functionality [161], a URL server [60], an annotation server [61], an electronic database [62], and a network content delivery [63] for the purpose of viewing the notepad. First, the user [59] visits any URL [186] in a web browser with notepad function [161], which requests the notepad Flavored Spanish Barley and Beans [187] from the annotation server [61], which requests the notepad object [188] from the electronic database data [62]. The electronic database [62] then returns the notebook object [189] to the annotation server [61], which returns the notebook [190] to the web browser with notepad functionality [161]. Next, the web browser with the notepad function [161] requests the annotation server [61] for a set of glycemic index annotation screenshots with annotation symbol data [191], which requests the glycemic index annotation object [192] from the electronic database [62], which returns the object annotation CDN URLs [193] to an annotation server [61], which returns a set of screenshots of the annotation CDN URLs and annotation symbol data [194] to a web browser with a notepad function [161]. Next, the web browser with the notepad feature [161] requests images from the CDN using a CDN URL [195] from the content delivery network [63], which returns images from the CDN for the specified URLs [196] back to the web browser with the notepad feature [161 ], which shows the displayed notepad [197] to the user [59]. The user [59] then clicks on the first displayed set of annotation screenshots [198] in a web browser with a notepad function [161], which requests a web document [199] from the server at URL [60], which returns a web document [200] back to a web browser with a notepad function [161], which shows the annotation symbol displayed in an open state on the web document [201]. A web browser with a notepad function [161] with annotation capability then displays the web document with the notepad on the side with the annotation symbol displayed in the open state and the browser scrolling to the appropriate location [202] to the user [59]. In the exemplary embodiment shown in FIG. 26 is a notepad creation flowchart showing the steps performed by a set of web browser extension commands that in an exemplary embodiment converts a web browser that supports adding browser extensions into a web browser with notepad functionality. The user [59] opens the browser [203] and then the web browser [102] downloads the browser extension [204]. User [59] then visits URL [205]. The web browser [102] then downloads the data sent from the visited URL [206] and injects the content script [207]. The user [59] then browses the web page [208] and creates a notebook [209]. As a result, the web browser extension content script [103] creates a notebook [210], and then the web browser extension background script [104] sends a request to the annotation server to create the notebook [211]. Next, the web browser extension content script [103] shows the displayed notebook [212], the user [59] views the displayed notebook and adds the already created annotation to it [213]. The web browser extension content script [103] then adds the already created annotation to the notebook [214], and the web browser extension background script [104] sends a request to add a reference to the annotation object in the notebook to the annotation server [215].

In the exemplary embodiment depicted in FIG. 27 is a sketchpad view flowchart illustrating the steps performed by a set of web browser extension commands that, in an exemplary embodiment, converts a web browser that supports browser extensions into a web browser with notepad functionality. The user [59] opens the browser [216] and the web browser [102] downloads the browser extension [217]. The user [59] then visits any URL [218]. The web browser [102] then downloads the data sent from the visited URL [219] and injects the content script [220]. The user [59] then opens notepad [221] and the web browser extension content script [103] requests notepad [222]. Next, the web browser extension background script [104] requests the annotation server for a notepad containing annotation screenshot display batches and annotation symbol data for all relevant annotation objects in the annotation object link list [223], and the web browser extension content script [103] displays displayed notepad [224]. The user [59] then views the displayed notepad and clicks on the first displayed set of screenshots

- 21 047005 annotation shots [225] and the web browser extension content script [103] sends a message to the browser to visit the url [226]. The web browser [102] then visits the URL [227] and downloads the data sent from the URL [228]. Next, the web browser extension background script [104] queries the annotation server for additional information about the annotation object, such as responses to annotation comments [229], and receives additional information about the annotation object from the annotation server [230]. Next, the web browser extension content script [103] shows the annotation symbol display in the open state [231] and scrolls the page to the location of the annotation symbol display [232]. The user [59] then views the page scrolled to the location of the displayed annotation symbol in the open state [233].

In the exemplary embodiment depicted in FIG. 28, a system diagram showing the interaction of a first user [234], a second user [235], a third user [236], a first user's browser [237], a second user's browser [238], a third user's browser [239], an annotation server [ 61], content delivery network [63] and electronic database [62]. Users communicate with an annotation server [61], which transmits images to a content delivery network [63]. Users also communicate with the content delivery network [63] to obtain images and other files. Users do not communicate directly with the electronic database [62]. In this exemplary embodiment, the annotation server [61] is hosted on one computer system and the electronic database [62] is hosted on a second computer system. Placing the electronic database [62] on the same machine as the annotation server [61] will reduce the number of machines used. The issue that needs to be addressed is the number of concurrent users of the system. Once this issue is resolved, the potential size of the system can be determined. The content delivery network [63] in this exemplary embodiment is operated by a third party. A content delivery network (CDN) may be offered by a third party. Many third parties offer CDNs. These CDNs can be configured using a web interface, and files can be downloaded and accessed through an application programming interface (API). This API may be a data transfer (REST) interface. Configuring and using such a third party CDN based on that third party's documentation should be understandable to those skilled in the art.

In the exemplary embodiment depicted in FIG. 29 is a component diagram showing how a more robust system can be built to isolate the various components. Separation of concerns can be beneficial to improve security and facilitate repeatable deployment of software across a network of machines. Additional security, separation of concerns, and repeatable deployment can make it easier to scale a system to handle more users. In fig. 29 shows a simple version of an annotation server [61] and an electronic database [62] interacting with each other, next to an alternative example that shows an annotation server [61] including a load balancer [241], an external web server [242] , Annotation API Server [243], Deep Learning API [244], and Message Broker [245]. Also shown is an electronic database [62], including a relational database [246] and a key value store database [247]. In this example implementation, a load balancer [241], a front-end web server [242], an annotation API server [243], a deep learning API [244], a message broker [245], a relational database [246], and a key store database values [247] each operate on a separate machine with one or more central processing units (CPUs), memory, and non-transitive computer readable media. Additionally, the deep learning API [244] has at least one of a graphics processing unit (GPU), a tensor processing unit (TPU), or a field programmable array (FPGA). In the exemplary embodiment shown in FIG. 29, the load balancer machine [241] has one central processing unit (CPU) with 4 cores, 8 GB random access memory (RAM) and a 250 GB solid state drive (SSD), and the front-end web server [242] has one CPU with 4 cores, 8 GB RAM and 250 GB SSD. The annotation API server [243] has two central processing units with 32 cores in each processor, for a total of 64 cores, 64 GB of RAM, and a 2 TB SSD. The deep learning api machine [244] has one CPU with 32 cores, 128 GB of RAM, two GPUs with 11 GB of RAM each and 2 TB SSD. The message broker machine [245] has one CPU with 4 cores, 32gb of RAM and 1tb SSD. The relational database machine [246] has one processor with 32 cores, 128 GB RAM and 2 TB SSD, and the key value store database machine [247] has one processor with 4 cores, 128 GB RAM and 1 TB SSD. Machines that meet these hardware specifications will allow this system to support multiple users with multiple concurrent users.

In the exemplary embodiment depicted in FIG. 29, hardware specifications have been described. The rationale behind these specifications will now be discussed. The load balancer [241] simply routes requests to the appropriate location. As a result, this machine's non-volatile machine-readable storage needs must be met by a 250GB SSD. Compared to the computation performed by the rest of the system, the load balancer [241] has a lower computational load and

- 22 047005 less memory load. As a result, 2 GB per core is enough, and 4 cores are enough for the rest of the system. As the system scales, processor cores executing more instructions per second and more processor cores can help the load balancer [241] avoid becoming a bottleneck. However, the load balancer [241] is unlikely to be a bottleneck given the characteristics shown for the example implementation depicted in FIG. 29. An external web server [242] also does not require a large amount of non-volatile machine-readable data storage space, a large number of processor cores, or a large amount of memory. This machine does not store user data, and the frontend code it stores can be supported by dedicated memory space. Moreover, the processor cores of the front-end web server machine [242] simply pass the requested resources to the user, and the user does not request the front-end code as often as the user's browser makes api calls to the annotation API server [243] during normal use. As a result, the processor cores for the front-end web server [242] machines have lower computational load and memory requirements than the annotation API server [243] machines. The Annotation API Server [243] has a heavy computational load because API requests are made by users' web browsers during the creation, organization, and viewing of annotations. The overall processing power of the annotation API server [243] can become a bottleneck when there are a large number of concurrent users. In an approximate version, this machine has 64 processor cores and 64 GB of RAM. If the annotation API server code [243] is written in a thread-safe manner, 1 GB of RAM per processor core is sufficient. If the code is not thread-safe or the web framework is memory intensive, it may be worth using 2 GB of RAM per processor core for the annotation API server [243]. To scale the annotation API server [243] to support more users, it is more cost effective to add more machines than to equip each machine with more processor cores and more RAM. The hard disk size for the Annotation API Server [243] is primarily needed to store log files. The Deep Learning API [244] can benefit from using 2 GPUs with enough GPU RAM to run deep learning models. CPU cores and RAM provide intensive data processing on the CPU and transfer of data to memory for GPU consumption. The 2TB SSD can store large deep learning models and other data for those models. The message broker [245] must run every time the system is running, but it will not lack computing power. 32 EB RAM is present to provide enough RAM to store large messages that may contain images. Since these images are not stored for a long time, 32 EB of RAM is sufficient. In addition, 4 cores are sufficient to operate the message broker [245]. The message broker [245] has lower computational processing requirements in terms of instructions per second than the annotation API server [243] in this example implementation. The key value store database [247] in this exemplary embodiment uses a single core. However, the key value store database [247] uses main memory to store in memory the reputations of various objects, such as annotation objects, and user votes for these objects. In addition, disk space is used for logging and to temporarily store backup copies of the key value store database in memory. The relational database [246] has 128 EB of RAM so that all data in the database can be stored in RAM while running, has a 2 TB SSD to provide sufficient memory for logging, and has 32 cores to provide computing capabilities to process requests sent by the annotation API server [243].

To scale up the system depicted in the exemplary embodiment of FIG. 29, the following steps can be taken. The new example design would have one or more load balancers [241], one or more front-end web servers [242], one or more annotation API servers [243], and one or more deep learning apis [244]. Load balancers, front-end web servers, annotation API servers, and deep learning API servers can be added to improve the performance of the entire system. Often message brokers can be organized into a message broker cluster. A message broker cluster can be used to scale the new exemplary implementation. Most key value store databases [247] also have the ability to create clusters of key value store databases.

In order for the relational database [246] to serve the largest number of users in the new exemplary embodiment, it is ideal to use a relational database [246] that uses threads rather than processes. In some cases, threads take up about 256 KB of memory per thread, while processes take up 10 MB of memory per process. This allows relational databases with thread-based connections to have 20 times more open connections than relational databases with process-based connections. Even when using the connection pooling facilities of relational databases, which rely on processes for open connections, thread-based connections are a better fit for style

- 23 047005 connections created by the annotation API server [243]. The Annotation API Server [243] relies on transactions and has performance needs. As a result, it is ideal to use a SQL transaction pool rather than a SQL session pool or a SQL statement pool. SQL session aggregation suffers from poor performance, and SQL transaction aggregation makes transactions more complex to implement than using SQL transaction aggregation. When using a SQL transaction pool, thread-based connections provide the best performance when computer statements must be executed on a processor not used by the relational database [246] because the connection remains open while work is performed on an external processor. As a result, the relational database [246] may be given fewer SQL queries to execute than the relational database [246] can process. This results in a decrease in overall throughput.

To further expand the capabilities of a relational database [246], it is beneficial to scale the relational database [246] by creating readable copies. The system can be scaled even further by reducing the write load on the relational database [246]. The write load can be reduced by implementing reputation voting through a key value store database [247] in a new exemplary embodiment. To further increase the capacity of the relational database, the number of processor cores and RAM of the relational database engine [246] can be increased in a new exemplary embodiment. Another step that can be taken is to use an in-memory relational database [246], which implements SQL without the full durability guarantees of most relational databases. Such systems may use an in-memory database with checkpointing and write-ahead logging to ensure that database data other than that entered into the in-memory database within the last few seconds is not lost in the event of a power loss. Beyond these steps, a relational database can be further scaled using... a high performance computing (HPC) cluster with remote direct memory access (RDMA) over an Infiniband network to allow the relational database to utilize memory beyond what is available locally on one car. Since this HPC cluster can be scaled up to hundreds or thousands of machines, the corresponding relational database [246] can support a large number of concurrent users.

In the exemplary embodiment depicted in FIG. 30 shows various types of client electronic devices [248] that can communicate with an annotation server [61] and a content delivery network [63]. Shown are Laptop [249], Desktop [250], iPad [251], Android Tablet [252], iPhone [253], Android Phone [254], Apple TV via Air-Web [255], Android TV [256], and Raspberry Pi [257]. Laptop [249], Desktop [250], Android Tablet [252], Android Phone [254], and Raspberry Pi [257] support a web browser with annotation capability and notepad functionality by installing a web browser and a web browser extension. Yandex Browser provides a web browser with support for web extensions on Android devices such as Android tablet [252] and Android Phone [254]. iPad [251] and iPhone [253] support a web browser with annotation capabilities through the installation of the Safari web browser and app extension. Apple TV via AirWeb [255] and Android TV [256] may not currently support the installation of a web browser with annotation capability, including a web browser and web browser extension. Additionally, the form factor may not be ideal for annotation. However, you can create progressive web applications for these devices that interact with the annotation server. This will allow you to display annotated material in a similar way to how a digital photo frame displays photographs. This will allow these devices to display multiple annotations created by other devices shown in FIG. thirty.

Annotations of images, videos, and audio recordings will now be covered. Graphical user interface diagrams of exemplary embodiments will be presented showing the end result of what the systems display and non-transferable computer readable media instructions executed in a web browser running on the processor. In exemplary embodiments of the invention, the instructions on the non-transitory computer-readable medium may be targeted to web browsers running on the processor, or the instructions on the non-transitory computer-readable medium may be included in the web browser with annotation capability. As previously discussed, the annotation-enabled web browser in one exemplary embodiment is a web browser with a web browser extension. When you use a web browser with annotation capabilities, you can annotate images, videos, and audio embedded in any web page. The website being annotated does not need to contain machine-readable instructions for annotating embedded images, videos, and audio recordings. When using a conventional web browser, images, videos and audio recordings can be annotated if instructions on a non-transitory machine-readable medium are transmitted through a specific website and the annotation of the content occurs on that specific site.

In addition to annotation-capable web browsers that annotate documents and media on the World Wide Web, software applications for mobile devices and Internet TVs can also use instructions on non-volatile machine-readable media that implement techniques

- 24 047005 methods and interfaces described in this document for annotating images, videos and audio recordings available to a group of users on the Internet or local intranet.

After discussing the GUI, we'll look at datasets created to transfer data between components. After discussing the relevant data sets, the methods and system architecture will be discussed in the context of the system architecture and components that have already been discussed.

In the exemplary embodiment depicted in FIG. 35, a snapshot [299] is shown with an annotation subject [300], an annotation subject bounding box [301], an annotation arrow [302], and a boxed annotation comment [303]. An annotation object [300] is an object of interest in an image. The annotation object [300] may be a person, car, soccer ball, tennis ball, or other object that can be identified in the image by a person or computer system. A bounding box [301] is a bounding box drawn around an annotation object [300] by a user or computer system. The annotation comment in field [303] is created by the user, and the annotation comment must refer to the annotation object [300]. An annotation arrow [302] is a line, which may have arrowheads on either side, that connects the bounding box [301] and the boxed annotation comment [303]. The Annotation Subject Border [301], Arrow [302], and Annotation Box Comment [303] were created by the user and are objects included in the image annotation. The user can select the rectangle tool from a panel located on a web page or a floating toolbar and draw a subject annotation box on the figure [301]. In addition, the user can enter text and place a comment on the picture annotation in the [303] field. The annotation subject boundary [301] and the annotation comment in the field [303] are connected by an annotation arrow [302]. Drawing annotation objects may not all be visible at the same time. For example, only the bounding box [301] may be visible, and the rest of the annotation objects become visible after the user hovers the mouse over the annotation object [300] or bounding box [301]. Visible annotation elements are those elements that are visible without the need for the mouse to hover over any other element. Mouse-over annotation elements are elements that become visible when the user hovers the mouse over visible annotation elements.

Below the image [299] there is a control panel that includes a button with an arrow for the previous annotation [304], a button with an arrow for the next annotation [305], a button for enabling collisions [306], a button for adding an image to a candidate annotation group [307], and a group list window Candidate Annotations [308] and the Create Annotation Group button [309]. The button label for adding an image to an annotation candidate group [307] has the shorter name Add to annotation group. However, the picture annotation is first added to the candidate annotation group, which then becomes the picture annotation group after the user clicks the create annotation group button [309]. The picture annotation includes objects such as a bounding box image [301], an arrow image [302], and a picture box image [303]. An image may contain more than one image annotation object [300]. In an electronic database [62], picture annotations are stored as picture annotation objects. These image annotation objects are then assigned by an electronic database [62], which gives a sequential order to the image annotations. This ordering of picture annotations gives meaning to the concept of previous and next picture annotations. The Previous Annotation Arrow button [304] removes the features included in the current annotation, provided the current annotation is not a candidate annotation, from the display view, and then adds the features of the previous annotation to the display view. Arrow button [305] removes the objects included in the current image annotation, if the current image annotation is not a candidate image annotation, from the display view, and then adds the objects of the next annotation to the display view. An image annotation group includes multiple image annotations. Before you can create an image annotation group, you must select image annotations. During the selection process, image annotations become candidate annotations. Candidate annotations are listed in the Candidate Annotation Group list box [308]. In fig. 35 shows a scenario in which image annotations have not yet been added to the candidate image annotation group list [308]. Thus, the message in the candidate group list field of annotation [308] is an empty candidate group list message in annotation [310].

The Enable Collisions button [306] defines the display properties of image annotations that collide with candidate image annotations listed in the Candidate Image Annotation Group list [308]. Possible states include collision exclusion and collision inclusion. If you click the Enable Collisions button [306], the image annotations that collide with the candidate image annotations listed in the Candidate Image Annotation Group list box [308] will be displayed. In the exemplary embodiment shown in FIG. 35, the button label will change to Avoid Collisions. If you click this button again, the image annotations that collide with the candidate image annotations listed in the annotation group list

- 25 047005 candidate image notations [308], will not be shown. If you click the [309] Create Annotation Group button, an item will be created that displays an annotation group that includes all the annotations listed in the list box [308] at the time you clicked the [309] Create Annotation Group button.

In alternative embodiments, the picture annotation [301] and the picture annotation arrow [302] may not be displayed. In an alternative embodiment, only a bounding box [301] and a boxed annotation comment [303] are present. Or, in another alternative embodiment, next to the annotation object [300] there is only a comment for the annotation in the field [303].

In the exemplary embodiment shown in FIG. 36, a picture [299] is shown with the first element of the annotation candidate group list [311] added to the annotation candidate group list field [308]. This state occurs when the add picture button to the annotation candidate group [307] is clicked in the state of the exemplary embodiment shown in FIG. 35. After adding a picture annotation to the annotation candidate group list, the state of the exemplary embodiment shown in FIG. 37 is created by pressing the next picture annotation arrow button [305] in the exemplary state of FIG. 36.

In the exemplary embodiment depicted in FIG. 37, a second image annotation object [312] is shown along with a user-created second image annotation object bounding box [313], a second image annotation arrow [314], and a second image annotation comment field [315]. If in the state of the exemplary embodiment shown in FIG. 37, click the add image button [307] to the candidate annotation group, the state of the exemplary embodiment shown in FIG. 38.

In the exemplary embodiment depicted in FIG. 38, the second element of the candidate image annotation group list [316] is shown. If the Create Annotation Group button [309] is clicked, an annotation group will be created including the two annotations shown in the Candidate Annotation Group list box [308] in FIG. 38.

In the exemplary embodiment depicted in FIG. 39, the user previously clicked the Enable Collisions button [306] in FIG. 36, which made it possible to show colliding picture annotations. After pressing the next annotation arrow button [305] from the state shown in FIG. 36, the condition shown in FIG. 39. In FIG. Figure 39 shows the arrow annotation [317] and field comment [318] for the colliding image annotation.

In the exemplary embodiment depicted in FIG. 40, video [319] is shown playing back video annotation [320]. The video [319] shows the video annotation object [321], bounding box [322], arrow [323], and boxed comment [324]. An annotation object [321] is an object of interest in a video. The annotation object [321] can be a person, car, soccer ball, tennis ball, or other object identifiable in the video by a person or computer system. An annotation object bounding box [322] is a bounding box drawn around an annotation object [321] by a user or computer system. The annotation comment in field [324] is created by the user, and the annotation comment must refer to the annotation object [321]. A video annotation arrow [323] is a line, which can have arrow heads on either side, that connects the bounding box of the video annotation object [322] and the boxed video annotation comment [324]. The boundary area [322], arrow [323], and box comment [324] were created by the user and are objects included in the video annotation. The user can select the rectangle tool from a panel located on a web page or a floating toolbar and draw an initial bounding box for the video annotation object [322]. In addition, the user can enter text and place a comment on the video annotation in the field [324] and connect them with an arrow [323]. As the video [319] plays, the video annotation [320] also plays. During playback of the video annotation [320], the user creating the video annotation can move the bounding box [322] of the video annotation object using the button [323].

mouse to ensure that the video annotation object bounding box [322] is positioned correctly throughout the video. The user can move the video annotation comment in the [324] box using the mouse to ensure that the video annotation comment in the [324] box is positioned correctly throughout the video. Once the motion trajectories of the bounding box [322] and the box comment [324] are known, the motion path of the arrow [323] along with its time rotation, time contraction, time expansion, and time position can be calculated according to trajectories of movement of the bounding box [322] and the comment in the box [324]. The user can also pause the video and resize the video annotation object bounding box [322] and the video annotation comment box [324]. Video annotation objects may not all be visible at the same time. For example, it is possible that only the bounding box [322] of a video annotation object will be visible, and the remaining video annotation objects will become visible after the user moves the mouse over the video annotation object [321] or the bounding box [322] of a video annotation object. Visible video annotation elements are those elements that are visible without the need to hover over any other element. Video annotation mouseover elements are elements that become visible when the user hovers the mouse over visible

- 26 047005 video annotation elements.

Below the video [319] there is a control panel including a previous annotation arrow button [325], a next annotation arrow button [326], a collision button [327], an add to annotation group button [328], a candidate annotation group list window [ 329] and the create annotation group button [330]. The label for the button to add a video to an annotation candidate group [328] has the shorter name Add to annotation group. However, the video annotation is first added to the candidate annotation group, which then becomes the video annotation group after the user clicks the create annotation group button [330]. Video annotation includes objects such as bounding box [322], arrow [323], and boxed comment [324]. A video annotation can contain more than one subject of the video annotation [321]. In an electronic database [62], video annotations are stored as video annotation objects. These video annotation objects are assigned a sequential ordering by the electronic database [62], which gives a sequential ordering to the video annotations. This ordering of video annotations gives meaning to the concept of previous and next video annotations. Previous Video Annotation Arrow Button [325] removes objects included in the current video annotation from the display view, provided that the current video annotation is not a candidate video annotation, and then adds objects from the previous video annotation to the display view. Arrow button [326] removes from the display view the objects included in the current video annotation, assuming that the current video annotation is not a candidate video annotation, and then adds the objects of the next video annotation to the display view. A video annotation group includes multiple video annotations. Before you can create a video annotation group, you must select video annotations. During the selection process, video annotations become candidate video annotations. The candidate video annotations are listed in the candidate video annotation group list box [329]. In fig. 40 shows a scenario where video annotations have not yet been added to the candidate video annotation group list [329]. Thus, the message in the Video Candidate Annotation Group List field [329] is an empty Video Candidate Annotation Group List message [331].

The Videos Include Collisions button [327] defines the display properties of video annotations that collide with candidate video annotations listed in the Candidate Video Annotation Group List box [329]. For video annotations, a collision means that two video annotations collide at some point in time. The collision may be shown in red in the video annotation playback line [320]. If you click the Enable Video Collisions button [327], the video annotations that collide with the candidate video annotations listed in the Candidate Video Annotation Group List [329] will be displayed. In the exemplary embodiment shown in FIG. 40, the button label changes to Avoid Collisions. If you click this button again, video annotations that collide with candidate video annotations listed in the candidate video annotation group list [329] will not be shown. If you click the Create Video Annotation Group button [330], an item will be created displaying a video annotation group that includes all the video annotations listed in the Video Annotation Candidate Group List window [329] at the time you clicked the Create Video Annotation Group button [330]. If the add video to annotation candidate group button [328] is pressed in the state shown in the exemplary embodiment of FIG. 40, the state shown in the exemplary embodiment of FIG. 41, wherein the first element of the annotation candidate group list [335] is added to the annotation candidate group list field [329].

In alternative embodiments of the invention, the limit box [322] and arrow [323] may not be displayed. In an alternative embodiment, only the bounding box [322] and the video annotation comment in the box [324] are present. Or, in another alternative embodiment, next to the video annotation object [321] there is only a video annotation comment in the field [324].

In the exemplary embodiment shown in FIG. 40, video player controls [319] are separate from annotation playback controls [320]. Alternatively, you can combine both controls into a single playback control. This may be a more complex input interface, but it reduces the number of elements displayed on the screen. This note about the integration of two playback controls also applies to the exemplary embodiments depicted in FIG. 41, fig. 42, fig. 43, fig. 44 and fig. 45.

In the exemplary embodiment depicted in FIG. 41 shows the motion path of the video annotation subject bounding box [332], the motion path of the video annotation arrow [333], and the motion path of the video annotation comment input box [334]. The Video Annotation Subject Bounding Box Motion Trajectory [332] shows the motion trajectory of the Video Annotation Subject Bounding Box [322], which should serve as the bounding box for the Video Annotation Subject [321] as the Video Annotation Subject [321] moves through the video. Video Annotation Arrow Trajectory [333] shows the trajectory of a video annotation arrow [323] as the video annotation arrow [323] moves through time along the video [319]. Annotation arrow [323] moves, rotates, expands and contracts over time since the bounding box

- 27 047005 annotation subject [322] and the annotation comment in field [324] moves in time. The trajectory of the video comment input field [334] shows the movement trajectory of the video comment in the field [324], since the video comment in the field [324] moves as the video [319] advances in time.

In the exemplary embodiment depicted in FIG. 42, a second video annotation object [336], a bounding box of a second video annotation object [337], a second video annotation arrow [338], a second video annotation boxed comment [339], a motion path of a second video annotation object [340], a first motion path of a second video annotation arrow are shown. [341], the second arrow path of the second video annotation [342], the final bounding box location of the second video annotation object [343], and the final arrow location of the second video annotation [344]. The boundary of the second annotation object [337] follows the second annotation object [336] along the path of the second annotation object [340] to the final location of the second annotation object [343]. The second video annotation arrow [338] transforms to the final location of the second video annotation arrow [344] over time. The transformation, including movement, rotation and compression, is depicted in the first arrow path of the second video annotation [341] and the second arrow path of the second video annotation [342]. The second video annotation comment in field [339] remains stationary in time. The second annotation candidate group list item [345] is displayed in the annotation candidate group list box [329], which occurs when the user clicks the add video to annotation candidate group button [328].

In the exemplary embodiment depicted in FIG. 43, the user has previously pressed the video collision enable button [327] in the state depicted in the exemplary embodiment of FIG. 41. This will show video annotations that have collisions with video annotations in the candidate annotation group list. After clicking the next video annotation arrow button [326], after clicking the collision enable button [327], in the state shown in the exemplary embodiment of FIG. 41, a video annotation collision arrow [346] and a video annotation collision comment in the box [347] can be displayed. In the exemplary embodiment of FIG. 43 shows a collision. However, a collision may occur at a point not depicted in a particular video frame. In such cases, the collision may be depicted in red on the video annotation playback timeline [320] in an example embodiment.

In the exemplary embodiment depicted in FIG. 44, playback of a sound recording [348] and reproduction of an annotation of a sound recording [349] are shown. Shown below the audio annotation playback [349] are an audio annotation arrow [350], an audio annotation comment field [351], an audio annotation connection arrow [352], and a derived audio annotation comment field [353]. Below the sound recording annotation playback [349] is a control panel that includes an arrow button for the previous sound recording annotation [354], an arrow button for the next sound recording annotation [355], a button for adding a sound recording to an annotation candidate group [356], a button for creating a derivative sound recording annotation [356]. 357], the sound recording annotation candidate group list window [358], the sound recording annotation group creation button [359], and the derived sound recording annotation comment input field [360]. The button label for adding a sound recording to an annotation candidate group [356] has the shorter name Add to annotation group. However, the audio annotation is first added to the audio annotation candidate group, which then becomes an audio annotation group after the user clicks the create audio annotation group button [359].

To create an annotation for a sound recording in the exemplary embodiment shown in FIG. 44, the user selects a short segment of audio annotation playback [349] using the mouse. Once selected, a sound recording annotation arrow [350] and a sound recording annotation comment field [351] appear from the middle of the selection, allowing the user to create a sound recording comment in the sound recording comment field [351]. Audio annotations include objects such as the audio annotation arrow [350] and the audio annotation comment field [351]. A short portion of the audio recording annotation playback [349] may be marked as shown in FIG. 49 in the annotation playback range of the built-in audio recording [393]. Although in FIG. 49 shows an embedded audio recording and a marked short segment is in the playback panel for playing the audio recording [348], the same style of marked segment can be used to play the audio recording annotation [349]. A segment can also be left unchecked, or a marked segment can appear when the mouse cursor hovers over a selected segment, a sound recording annotation arrow [350], or a sound recording annotation comment field [351]. Additionally, a question audio annotation symbol may be introduced, which behaves similar to the question audio annotation symbol [394] in FIG. 49.

An audio annotation can contain more than one selected audio segment. In an electronic database [62], sound recording annotations are stored as sound recording annotation objects. These sound recording annotation objects are then assigned by an electronic database [62], giving the sound recording annotations a consistent order. This arrangement of sound recording annotations gives

- 28 047005 meaning of the concept of the previous and next annotations of the sound recording. The previous audio annotation arrow button [354] removes from the display view the objects included in the current audio annotation, provided that the current audio annotation is not a candidate annotation, and then adds the objects from the previous audio annotation to the display view. The next sound recording annotation arrow button [355] removes from the display view the objects included in the current sound recording annotation, assuming that the current sound recording annotation is not a candidate sound recording annotation, and then adds the objects of the next sound recording annotation to the display view. Audio annotation groups are a group of audio annotations that the user wants to group together. These groups are displayed as a single element, and one or more audio annotations from a group of audio annotations can be used to create derived audio annotations associated with the one or more audio annotations. Additionally, a derived sound recording annotation, along with the sound recording annotations on which it depends, may be displayed as a single element. Sound recording annotations can be added to a sound recording annotation group by clicking the Add Sound Recording to Candidate Annotation Group button [356]. As a result, the new sound recording annotation will appear in the list of candidate sound recording annotation groups [358]. After adding audio annotations to an audio annotation group, the user can create derived audio annotations. Derived annotations are annotations that link other sound recording annotations and then add a comment to the derived sound recording annotation comment field [353] connected to multiple sound recording annotation comment fields. The derived annotation comment field [353] is connected to each sound recording annotation comment field [351] using the sound recording annotation connection arrow [352]. A derived audio annotation is created by selecting the desired audio annotations and clicking the Create Derived Audio Annotation button [357]. The user then enters the derived audio annotation comment [361] into the derived audio annotation comment input field [360] and presses the Enter key. The Sound Recording Annotation Candidate Group List field [358] contains a list of the elements of the Sound Recording Annotation Candidate Group [362]. This list includes both sound recording annotations and derivative sound recording annotations. The first element in the sound recording annotation candidate group element list [362] is the first element in the sound recording annotation candidate group list [363].

In the exemplary embodiment depicted in FIG. 45, video playback [319] and video annotations [320] are shown. Below the video annotation playback [320] are shown a video annotation arrow [364], a video annotation comment field [365], a video annotation connection arrow [366], and a derived video annotation comment field [367]. Below the video annotation playback [320] is a control panel that includes a previous annotation arrow button [325], a next annotation arrow button [326], a collision button [327], a button for adding video to a candidate annotation group [328], a create button Derived Annotation [368], Candidate Annotation Group List Window [329], Create Annotation Group Button [330], and Derived Annotation Comment Entry Field [369]. The label for the button to add a video to an annotation candidate group [328] has the shorter name Add to annotation group. However, first the video annotation is added to the video.

A candidate for an annotation group that subsequently becomes a video annotation group after the user clicks the create annotation group button [330]. To create a video annotation, in an exemplary embodiment, the user selects a short video annotation playback segment [320] using the mouse. Once highlighted, an arrow [364] and a comment box [365] appear from the middle of the selection, allowing the user to comment on the video in the comment box [365]. A short video annotation playback segment [320] may be marked as shown in FIG. 46 built-in video annotation playback range [375]. Although in FIG. 46 shows an embedded video and a tagged short segment is in the video playback panel [319], the same style of tagged segment can be used to play a video annotation [320]. A segment can also be left unlabeled, or a labeled segment can appear after hovering the mouse over a selected segment, a video annotation arrow [364], or a video annotation comment field [365]. Additionally, a question video annotation symbol may be introduced that behaves similar to the question video annotation symbol [376] in FIG. 46.

As previously stated in another exemplary embodiment, video annotations include objects such as a bounding box [322], an arrow [323], and a boxed comment [324]. A video annotation can contain more than one subject of the video annotation [321]. In an electronic database [62], video annotations are stored as video annotation objects. These video annotation objects are assigned a sequential ordering by the electronic database [62], which gives a sequential ordering to the video annotations. This ordering of video annotations gives meaning to the concept of previous and next video annotations. Previous Video Annotation Arrow Button [325] removes objects included in the current video annotation from the display view, provided that the current video annotation is not a candidate video annotation, and then adds it to the display view

- 29 047005 lenie objects of the previous video annotation. Arrow button [326] removes from the display view the objects included in the current video annotation, assuming that the current video annotation is not a candidate video annotation, and then adds the objects of the next video annotation to the display view. Video Annotation Groups are a group of video annotations that the user wants to group together. These groups are displayed as a single element, and one or more video annotations from a group of video annotations can be used to create derived video annotations associated with those one or more video annotations. Additionally, the derived video annotation, along with the video annotations on which it depends, can be displayed as a single element. Video annotations can be added to a candidate video annotation group by clicking on the add video to candidate annotation group button [328]. As a result, the new video annotation will appear in the list box of the candidate video annotation group [329]. Once video annotations are added to a candidate annotation group, the user can create derived annotations. Derived annotations are annotations that link other video annotations and then add a comment to a derived annotation comment field [367] connected to multiple video annotation comment fields. A derived annotation comment block [367] is connected to each video annotation comment block [365] using a video annotation connection arrow [366]. A derived video annotation is created by selecting the desired video annotations and clicking the create derived video annotation button [368]. The user then enters a derived annotation comment [370] in the Derived Annotation Comment input field [369] and presses Enter. The candidate video annotation group list field [329] contains a list of the elements of the candidate video annotation group [371]. This list includes both video annotations and derived video annotations. The first element in the candidate video annotation group element list [371] is the first element in the video annotation candidate group list [372]. The create video annotation group button [330] can be clicked to create an annotation group including the video candidate annotations in the video candidate annotation group list box [329]. The video collision enable button [327] operates in the same way as the video collision enable button [327] in the exemplary embodiments depicted in FIG. 40, fig. 41, fig. 42 and fig. 43. [0167] In the exemplary embodiment depicted in FIG. 46, embedded video [373] and embedded video playback [374] are shown. Embedded video [373] and embedded video playback [374] exist on a web page that is viewed using an annotable web interface.

browser. In the exemplary embodiment shown in FIG. 46, the embedded video annotation playback range [375], the embedded video annotation question symbol [376], the embedded video annotation disagree symbol [377], the embedded video annotation beacon [378], the embedded video annotation question symbol [379], the embedded video annotation disagree symbol [380], and embedded video annotation beacon playback time [381]. The Character Beacon Enable button [57] triggers the display of one complete video annotation beacon for each video annotation in the embedded video [373], unless the embedded video [373] contains more than 3 video annotations. If more than 3 video annotations are present, alternative display techniques may allow the user to better visualize the video annotations being viewed. The complete displayed video annotation beacon includes the embedded video annotation playback range [375], the embedded video annotation symbol, the embedded video annotation beacon [378], the embedded video annotation beacon symbol, and the embedded video annotation beacon playback time [381]. The embedded video annotation playback range [375] highlights the time range that the user believes is relevant to the video annotation. Embedded video annotation symbols, such as the question symbol [376] and the disagreement symbol [377], convey the semantic relationship between the selected video segment and the video annotation comment. The built-in video annotation beacon [378] serves to draw the user's attention to the existence of a video annotation and some of its properties. The embedded annotation beacon symbol places the semantic relationship between the selected video segment and the annotation comment in close proximity to the embedded annotation beacon [378]. Two examples of embedded annotation beacon characters are the embedded annotation beacon question character [379] and the embedded annotation beacon disagree character [380]. The playback time of the embedded annotation beacon [381], which is placed near the embedded annotation beacon [378], conveys the length of the video segment associated with the annotation comment.

In the exemplary embodiment depicted in FIG. 46, Embedded Video Annotation Playback range [375] and embedded video annotation symbol [376] are shown in the context of embedded video [373] and embedded video playback [374]. However, the video annotation reproduction range and the video annotation symbols may also be used when reproducing the video annotation [320] shown in FIG. 45.

In the exemplary embodiment depicted in FIG. 47, the embedded picture [382] is shown. An inline image [382] exists within a web page, and an inline image annotation object [383] exists within an inline image [382]. In the exemplary embodiment shown in FIG. 47, showing bounding box [384], arrow [385], and boxed comment

- 30 047005

[386] . An embedded image annotation object [383] represents an object of interest in the image. The annotation object of an embedded image [383] may be a person, a car, a soccer ball, a tennis ball, or another object that can be identified in the image by a person or a computer system. An annotation object bounding box [384] is a bounding box drawn around an annotation object [383] by a user or computer system. The user can select the rectangle tool from a panel located on a web page or a floating toolbar and draw a bounding box [384] for the embedded image annotation object. In addition, the user can enter text and place a comment on the annotation as an embedded image [386]. The inline image annotation comment in field [386] is user-generated, and the annotation comment must refer to the inline image annotation object [383]. An embedded image annotation arrow [385] is a line, which may have arrow heads on either side, connecting the embedded image annotation subject bounding box [384] and the embedded image annotation comment in a box [386]. The annotation subject boundary [384], annotation arrow [385], and annotation comment [386] were created by the user and are embedded objects included in the image annotation.

Embedded image annotation objects may not all be visible at the same time. As an example, it is possible to have only the inline drawing annotation object

The bounding box [384] to be visible, and other embedded annotation objects become visible after the user hovers over the annotation object [383] or the annotation object bounding box [384]. Visible image annotation elements are those inline elements that are visible without the need to hover over any other element. Inline picture annotation hover elements are inline elements that become visible when the user hovers the mouse over visible inline picture annotation elements. When you hover your mouse over a drawing's built-in annotation objects, a control panel for the contextual annotation option appears [387]. The user can use the contextual annotation control panel [34], built into the contextual annotation control panel [387], to change privacy settings, share via email, share on social networks, mark an annotation as inappropriate, delete an annotation object from an electronic database [62 ] if you have sufficient rights, or edit the built-in objects that make up the annotation for the picture. Additionally, the user can use the reputation selector [33] to vote on the reputation of an image annotation.

In the exemplary embodiment depicted in FIG. 48, embedded video [373] is shown. Embedded video [373] and embedded video playback [374] exist on the web page, and the embedded video annotation object [388] resides inside the embedded video [373]. In the exemplary embodiment shown in FIG. 48, a bounding box [389], an arrow [390], and a comment box [391] are shown. An embedded video annotation object [388] represents an object of interest in a video. The object of an embedded video annotation [388] can be a person, a car, a soccer ball, a tennis ball, or another object that can be identified in the video by a person or a computer system. A bounding box [389] is a bounding box drawn around an annotation object [388] by the user or computer system. As the embedded video annotation object [388] moves through the video, the embedded video annotation object's bounding box [389] must continue to act as a bounding box for the embedded video annotation object [388]. The embedded video annotation comment in field [391] is created by the user, and the annotation comment must refer to the embedded video annotation object [388]. An embedded video annotation arrow [390] is a line, which may have arrow heads on either side, connecting the embedded video annotation object bounding box [389] and the embedded video annotation comment in a box [391]. The embedded video annotation object boundary [389], the embedded video annotation arrow [390], and the embedded video annotation field comment [391] were created by the user and are embedded objects included in the video annotation.

The user can select the rectangle tool from a panel on the web page or a floating toolbar and draw the initial boundary of the embedded video annotation object [389]. Alternatively, the user can enter text and position the embedded video annotation comment in the field [391] and connect them with the embedded video annotation arrow [390]. To ensure that objects are positioned correctly during video playback, the user or computer system must define a motion path for the object bounding box of the embedded video annotation [389]. The user creating the video annotation is able to trace the trajectory of the embedded video annotation object's bounding box [389] as the video plays back over time. While the embedded video [374] is playing, the annotator can move the bounding box [389] with the mouse to ensure that the bounding box [389] is positioned correctly throughout the embedded video. The user can move the embedded video annotation comment in the [391] field using the mouse to ensure that the embedded com

- 31 047005 The video annotation commentary in the [391] field is correctly positioned throughout the video. A web browser, web browser extension, or web page code with annotation capabilities records the path of movement as the user drags the mouse. The next time the video is played back, the embedded annotation object [389] and embedded comment in the field [391] will follow their respective motion paths. Once the motion trajectories of the bounding box [389] and the embedded comment box [391] are known, the motion path of the embedded arrow [390] along with its time rotation, time contraction, time expansion, and time position can be calculated in according to the motion trajectories of the bounding box [389] and the embedded comment in the box [391]. If the size of the desired bounding box for the embedded video annotation object [388] changes while the video is playing, the user can pause the video and change the size of the embedded video annotation object bounding box [389] and the embedded video annotation comment box [391].

Embedded video annotation objects may not all be visible at the same time. For example, it is possible that only the bounding box [389] of an embedded video annotation object will be visible, and the rest of the embedded video annotation objects will become visible after the user drags the mouse over the embedded video annotation object [388] or the bounding box [389] of the embedded video annotation object. Visible video annotation elements are those embedded elements that are visible without the need to hover over any other element. Mouse-over video annotation inline elements are those inline elements that become visible when the user hovers the mouse over visible video annotation inline elements. When you hover your mouse over embedded video annotation objects, a control panel for the contextual annotation option appears [387]. The user can use the contextual annotation control panel [34], built into the contextual annotation control panel [387], to change privacy settings, share via email, share on social networks, mark a video annotation as inappropriate, delete a video annotation object from an electronic database [62 ] if you have sufficient rights, or edit the embedded objects that make up the video annotation. Additionally, the user can use a reputation selector [33] to vote on the reputation of a video annotation. Each relationship in a group of semantic annotation relationships has its own embedded video annotation symbol associated with it. For example, the embedded video annotation symbol Question [376] indicates that the creator of the video annotation has designated the annotation semantic relationship between the selected portion of the video and the video annotation comment as the relationship Question. The embedded video annotation question symbol [376] allows a user familiar with how the exemplary embodiment works to know that the user can click on the embedded video annotation symbol [376] and see a comment on the video annotation, which is a question about the playback range of the embedded video annotation [376]. 375] before the user actually performs the click action. The question annotation symbol, the disagreement annotation symbol, the agreement annotation symbol, the exclamation annotation symbol, and the advertising annotation symbol correspond to five relations in the group of semantic relations of the annotation. These semantic annotation relations are question relations, disagree relations, agreement relations, exclamation relations, and advertisement relations. These video annotation symbols and video annotation semantic relations are applied to the relationship between the embedded video annotation playback range [375] and the embedded video annotation comment in the field [391].

In the exemplary embodiment depicted in FIG. 49, the embedded audio [392] is shown with the embedded audio annotation playback range [393] and the embedded audio annotation symbol [394]. The embedded audio annotation playback range [393] indicates the selected portion of the audio recording corresponding to the audio annotation. Embedded audio annotation symbols are associated with elements of a group of annotation semantic relations. Each semantic annotation relation shows the relationship between the playback range of the embedded audio annotation [393] and the audio annotation comment. Each relation in a group of annotation semantic relations has its own embedded audio annotation symbol associated with it. For example, the embedded audio annotation symbol question [394] indicates that the creator of the audio annotation has designated the annotation semantic relationship between the selected portion of the audio recording and the audio annotation comment as a question relationship. The embedded audio annotation symbol [394] allows a user familiar with how the exemplary embodiment works to know that the user can click on the embedded audio annotation symbol [394] and see an audio annotation comment, which is a question about the playback range of the embedded audio annotation [394]. 393] before the user actually performs the click action. The annotation comment window [31] appears when you hover your mouse over the annotation playback range of the embedded audio recording [393]. The user can use the contextual annotation options control panel [34] built into the annotation comment field [31] to change privacy settings, share by email, share on social networks, mark an audio annotation as inappropriate, remove an audio annotation object from the electronic database data [62] if you have sufficient rights, or edit the embedded objects that make up the audio annotation. In addition, the use

- 32 047005 The body can use the reputation selector [33] to vote on the reputation of an audio annotation. The question annotation symbol, the disagreement annotation symbol, the agreement annotation symbol, the exclamation annotation symbol, and the advertising annotation symbol correspond to five relations in the group of semantic relations of the annotation. These semantic annotation relations are question relations, disagree relations, agreement relations, exclamation relations, and advertisement relations. These audio annotation symbols and audio annotation semantic relations refer to the relationship between the playback range of the embedded audio annotation [393] and the audio annotation comment [32].

Aspects of the present invention have been described in FIGS. 35, fig. 36, fig. 37, fig. 38, fig. 39, fig. 40, fig. 41, fig. 42, fig. 43, fig. 44, fig. 45, fig. 46, fig. 47, fig. 48 and fig. 49, and it will be appreciated that each element of these graphical circuits can be implemented using computer readable program instructions. The following describes methods and systems associated with the features and capabilities depicted in the example embodiments of FIGS. 35, fig. 36, fig. 37, fig. 38, fig. 39, fig. 40, fig. 41, fig. 42, fig. 43, fig. 44, fig. 45, ig. 46, fig. 47, fig. 48 and fig. 49 will be described and related to previously presented material. The description of these methods and systems includes a description of the sets of data elements used to receive and store data, as well as the sets of display elements used to transmit and display visual elements on client devices. As sets of display elements are described, various blocks of elements will be described. After describing these blocks of elements, the relationship of these elements to the other elements that have been presented, including the user interface, will be discussed. The data element sets, display sets, and element blocks that will be described are to be considered as descriptions in the context of the exemplary embodiment.

In the context of an exemplary embodiment of the invention, sets of data elements used to receive data from client devices and store that data using one or more computer systems will now be described. Data element sets are image annotation sets, video annotation sets, audio annotation sets, embedded image annotation sets, embedded video annotation sets, embedded audio annotation sets, annotation group sets, derived annotation group sets, and derived annotation group sets.

In the context of one exemplary embodiment of the invention, image annotation sets include one or more image annotation atoms. An image annotation atom is a unit that can be meaningfully used on its own to annotate an image. For example, an image annotation atom may include a bounding box [301], a boxed comment [303], and an arrow [302], as shown in the exemplary embodiment depicted in FIG. 36. However, you can also use the annotation comment in field [303] by itself next to the annotation object [300]. Thus, the picture comment in field [303] can itself be an atom of the picture annotation. In addition to the image annotation atoms, the image annotation set includes information about the frame of the image being annotated. Frame information includes the image size, specified by the number of pixels along the x-axis and the number of pixels along the y-axis. Once the image size is determined, the elements that make up the annotation atom, such as the bounding box [301], boxed comment [303], and arrow [302], are given relative coordinates based on the box information. If the boundary of an annotation item [301] is specified by the x and y coordinates of its center, width and length, then the coordinates are specified relative to the lower left corner of the image. The annotation comment in field [303] can be specified in a similar manner along with markup information for the annotation comment contained in field [303]. This markup information contains font size as well as text formatting information. In addition, information about the text generation mechanism is contained in the image annotation link. Engine creation can be user-written, image captions using artificial intelligence (AI) or deep learning models, other annotation, or a combination of methods. Additionally, at least one of a user name, a machine learning model, and other annotation data is recorded. If a combination of methods is used, this information is also included in the image annotation set. An image annotation arrow [302] may be specified by the x and y coordinates of its start and end points along with information indicating whether an arrow head is present at the start point and whether an arrow head is present at the end point. Again, the coordinates are relative to the lower left corner of the drawing. In addition to relative coordinate information, each element included in the annotation atom specifies a generation mechanism. The generation mechanism indicates whether the item was generated by a user, a machine learning model, or a simple algorithm. In addition, at least one of the username, the name of the machine learning model, and the name of the algorithm are recorded. In addition, ovals can be used. Ovals are specified by the x and y coordinates of the center relative to the lower left corner of the image, the width, height, and rotation of the oval. In addition, the generation mechanism is recorded, as well as at least one of the username, the name of the machine learning model, and the name of the algorithm.

- 33 047005

In the context of an exemplary embodiment, the built-in picture annotation sets contain all the information in the picture annotation bundle, as well as the url of the web page or document on which the picture resides. The description written for picture annotation sets should be interpreted to take into account that the embedded picture annotation subject [383] is replaced by the picture annotation subject [300], the embedded picture annotation subject bounding box [384] is replaced by the picture annotation subject bounding box [301] , the inline picture annotation arrow [385] is replaced with the picture annotation arrow [302], and the inline picture box annotation comment [386] is replaced with the box picture annotation comment [303]. Drawings of figs. 36 and fig. 47 should be considered in more detail. If the document is a pdf or complex web page, image search information such as the pdf document page or image DOM locators is also stored.

In the context of one exemplary embodiment of the invention, video annotation sets include one or more video annotation atoms. A video annotation atom is a unit that can be meaningfully used on its own to annotate videos. For example, a video annotation atom may include a bounding box [322], a bounding box path [332], a comment input field [324], a comment input field path [334], an arrow [323], and an arrow path [333], as shown in the exemplary embodiment depicted in FIG. 41. However, you can also use the video annotation comment in field [324] and the motion path of the video annotation comment input field [334] independently, while the video annotation comment in field [324] will be located next to the annotation object [321] in video playback time. Thus, the video comment field [324] and the motion path of the video comment input field [334] can themselves be video annotation atoms. In addition to the video annotation atoms, the video annotation package includes information about the frame of the video being annotated. The frame information includes the video size, specified by the number of pixels on the x-axis and the number of pixels on the y-axis. This should include the video area [319], not the video annotation playback area [320]. Once the video size is determined, the elements that make up the video annotation atom, such as the video annotation subject boundary [322], the video annotation subject boundary motion path [332], the video annotation comment input field [324], the video annotation input field motion path [334], the video annotation arrow [323] and video annotation arrow trajectory [333], obtain relative coordinates based on frame information. If the boundary area of an annotation object [322] is specified by the x and y coordinates of its center, width and length, then the coordinates are specified relative to the lower left corner of the video. The trajectory of an annotation object [332] should determine the evolution of the annotation object [322] over time. It determines the change in the x and y coordinates of its center, width and length over time. This specification is accomplished by specifying the x and y coordinates relative to the bottom left corner of the video at discrete times during the playback range of the video annotation, along with an interpolation mechanism between these discrete time steps. Since the relative x-coordinate, relative y-coordinate, width, and length are real numbers, the same method of storing the values at discrete points in time and specifying the interpolation method is sufficient for all four values. Possible interpolation methods include linear interpolation, spline curves, Bezier curves, and other methods. The video comment in field [324] and the motion path in the video comment input field [334] can be specified using a similar method, together with the comment markup information contained in the video comment in field [324]. This markup information contains font size as well as text formatting information. In addition, information about the text creation mechanism is included in the video annotation link. The creation engine can be written by the user, a video caption using an AI or deep learning model, other annotation, or a combination of methods. Additionally, at least one of a username, a machine learning model, and other annotation data is recorded. If a combination of methods is used, this information is also included in the video annotation set. The video annotation arrow [323] and the video annotation arrow path [333] may be specified by the relative x and y coordinates of the start and end points along with information indicating whether an arrow head is present at the start point and whether an arrow head is present at the end point. Specification of an arrow's trajectory in a video annotation [333] is accomplished by specifying the relative x and y coordinates of the start and end points at discrete times during the playback range of the video annotation, along with an interpolation mechanism between these discrete time steps. Again, the coordinates are relative to the bottom left corner of the video. In addition to the relative coordinate information, each element included in the annotation atom defines a generation mechanism. The generation mechanism indicates whether the item was generated by a user, an AI or deep learning model, another annotation, or a combination of methods. In addition, at least one of a user name, an AI or deep learning model name, and an algorithm name is recorded. In addition, ovals can be used. Ovals are defined by the x and y coordinates of the center relative to the bottom left corner of the video, the width, height, and rotation of the oval. In addition, upon generation, the mechanism along with at least one of a user name, an AI or deep learning model name, and an algorithm name are recorded. Video annotation symbol information can also be included

- 34 047005 chen for video annotation sets.

In the context of an exemplary embodiment, embedded video annotation sets contain all the information within the video annotation bundle, as well as the url of the web page or document on which the video resides. The description written for video annotation sets should be interpreted to reflect that the embedded video annotation subject [388] is replaced by the video annotation subject [321], the embedded video annotation subject bounding box [389] is replaced by the video annotation subject bounding box [322], the embedded video annotation arrow [ 390] is replaced by a video annotation arrow [323], the embedded video annotation comment in the field [391] is replaced by a video annotation comment in the field [324], the embedded annotation object path is replaced by the annotation object path [332], the embedded annotation arrow path is replaced by the annotation arrow path [333], the path of an inline annotation comment in a field is replaced by the path of an annotation comment input field [334], and the inline annotation symbol is replaced by an annotation symbol. For details, refer to the drawings of FIGS. and fig. 48. If the document is a pdf or a complex web page, video search information such as the pdf document page or video DOM locators is also stored.

In the context of one exemplary embodiment of the invention, sets of audio annotations include one or more audio annotation atoms. A sound recording annotation atom is a unit that can be meaningfully used on its own to annotate a sound recording. For example, a sound recording annotation atom may include a sound recording annotation reproduction range, a sound recording annotation symbol, and a sound recording annotation comment. However, you can only use the playback range of the audio annotation and the audio annotation comment. Thus, a sound recording annotation playback range and a sound recording annotation comment may themselves include a sound recording annotation atom. The sound recording annotation comment may have associated markup information for the sound recording annotation comment. This markup information contains font size as well as text formatting information. In addition, the audio recording annotation link contains information about the mechanism for creating the text. The creation mechanism can be written by the user, recognized using an AI or deep learning model, copied from another annotation, or a combination of methods. In addition, the creation mechanism is recorded along with at least one of a user name, an AI or deep learning model name, and another annotation link. When using a combination of methods, more than one type of source name is recorded. The playback range of the audio recording annotation can also be determined based on the audio search. Thus, the generation engine for the audio playback range must also be recorded along with at least one of a user name, an AI or deep learning model name, or other annotation comment. When using a combination of methods, more than one generation mechanism and source information are recorded. In addition, information about the audio recording frame is recorded to facilitate placement of audio annotation playback ranges on audio annotation playback controls. In the context of an exemplary embodiment, embedded sound recording annotation sets contain all of the information contained in the sound recording annotation bundle, as well as the url of the web page or document on which the sound recording is located. The description written for audio annotation sets is to be understood to mean that the embedded audio annotation playback range [393] is replaced by the audio playback range, the embedded audio annotation symbol is replaced by the audio annotation symbol, and the embedded audio comment is replaced by the audio commentary. Additionally, it should be understood that the audio embedded frame information is recorded along with a set of audio embedded annotations to facilitate placement of the audio embedded annotation playback range [393] on the audio embedded playback controls. For details, please refer to the drawings of FIGS. 44 and fig. 49. If the document is a complex web page, audio location information, such as audio DOM locators, is also stored.

In the context of one exemplary embodiment of the invention, sets of annotation groups include references to all annotations included in that particular annotation group. Additionally, information about whether annotations collide is stored along with metadata indicating when the annotation group was created and whether collision detection was performed by a user or an AI or deep learning model. At least one of the users and the AI or deep learning models performing the collision detection is also stored in the annotation group stack.

In the context of one example implementation, derived sets of annotation groups include references to all annotations included in that particular derived annotation group. Additionally, information about whether annotations collide is stored along with metadata indicating when the annotation group was created and whether collision detection was performed by a user or an AI or deep learning model. At least one of the users and the AI or deep learning models performing collision detection is also stored in

- 35 047005 pack of derived annotation group. A set of derived annotation groups contains one derived annotation along with references to all the annotations included in that particular annotation group. Annotation connection arrows can be determined from the annotation links present in the derived annotation group bundle. In this exemplary embodiment, there is only one derived annotation that provides all the information for constructing the annotation connecting arrows. In addition, the comment for the derived annotation is stored in the derived annotation group bundle. The markup information for the derived annotation comment is also preserved. This markup information contains font size as well as text formatting information. In addition, information about the text creation mechanism is contained in the derived annotation group bundle. The generation engine can be written by the user, generated by an AI or deep learning model, copied from another annotation, or a combination of methods. Additionally, at least one of a username, a machine learning model, and other annotation data is recorded. If a combination of methods is used, this information is also contained in the derived annotation, group copula.

In the context of an exemplary embodiment, a derived annotation annotation group includes references to all derived annotations and annotations present in that particular derived annotation annotation group. Additionally, information about whether annotations collide or not is stored along with metadata indicating when the annotation group with derived annotations was created and whether collision detection was performed by a user or an AI or deep learning model. At least one of the users and the AI or deep learning models performing the collision detection is also stored in the annotation group stack.

Some types of display kits and item boxes will be described below. These descriptions are considered in the context of an exemplary embodiment. In the context of an exemplary embodiment of the invention, display sets such as image annotation display sets, video annotation display sets, audio annotation display sets, built-in image annotation display sets, built-in video annotation display sets, built-in audio annotation display sets, embedded annotation group display sets will be described. , derived annotation group display sets, and annotation group display sets with derived annotation. In the context of an exemplary embodiment of the invention, the following element blocks will be described: an image annotation element block, a video annotation element block, an audio annotation element block, an embedded image annotation element block, an embedded video annotation element block, an embedded audio annotation element block, an annotation group element block, a block of elements of a derived annotation group and a block of elements of an annotation group with derived annotations. One of the purposes of display packages, in an exemplary embodiment, is to provide data for a web browser to display the appropriate block of elements associated with a given type of annotation or media element. Thus, the element blocks will be described in connection with the annotation element block [36] presented in the exemplary embodiment depicted in FIG. 9. In addition to the numbered elements in the annotation element block, the annotation element block also contains highlighted text, the element block annotation symbol, the annotation element block comment, and the element block URL. Once the various element cells have been described in the context of the exemplary embodiment, the data structures corresponding to the various display bundles can be illustrated. Also, for clarity, it is sometimes useful to refer to display sets. Display convolutions are the visual output created using display convolutions. This visual output is contained in a block of elements. In contexts such as a notepad, the displayed package may be present without the entire block of elements.

In an exemplary embodiment of the invention, the image annotation element block would include a separate context variant annotation control panel [41] and a separate context reputation selector [42], as would the annotation element block [36], as shown in the exemplary embodiment of the invention in FIG. 9. Instead of a screenshot with marked text [37], the Image Annotation Elements window will contain a thumbnail of the image along with the image annotation displayed on top of the thumbnail. Instead of a separate highlighted context annotation text [38], a separate displayed context question annotation symbol [39], and a separate context annotation comment [40], the image annotation element field will display the image annotation comment, the url will not be present for image annotation element blocks, but it will be present for inline image annotation element blocks. In order for a web browser to display these blocks of annotation elements, the annotation display package must contain a tagged thumbnail image with the annotation displayed on top of it, with objects such as a bounding box [301], a boxed annotation comment [303], and an annotation arrow [302] , modified and applied properly. In addition, the annotation display set must contain the annotation comment, username, and reputation information.

In one exemplary embodiment, the inline image annotation window will include

- 36 047005 a separate contextual variant annotation control panel [41] and a separate contextual reputation selector [42], as well as an annotation window [36]. Instead of a screenshot with the text highlighted [37], the embedded image annotation window will contain a thumbnail of the embedded image along with the image annotation displayed on top of the thumbnail. Instead of the highlighted text of a separate context annotation [38], a separate context question display annotation symbol [39], and a separate context annotation comment [40], the inline image's annotation element field will display the inline image's annotation comment, the url will match the url that includes the inline figure , which was annotated using Figure Annotation. In order for a web browser to display these blocks of inline image annotation elements, the inline image annotation display bundle must contain a tagged thumbnail image of the inline image with the image annotation displayed on top of it, with objects such as the inline image annotation object bounding box [384], an annotation comment framed inline image [386] and inline image annotation arrow [385] modified and overlaid properly. In addition, the inline image annotation display set must contain the inline image annotation comment, username, and reputation information.

In an exemplary embodiment of the invention, the video annotation element block would include a separate contextual variant annotation control panel [41] and a separate contextual reputation selector [42], as would the annotation element block [36], as shown in the exemplary embodiment of the invention in FIG. 9. Instead of a screenshot with marked text [37], the video annotation element window will contain a thumbnail of the video frame at some selected point in time along with a video annotation displayed on top of the thumbnail of the video frame at the same selected point in time. Additionally, the play button may allow the thumbnail video to be played, showing a portion of the thumbnail video and video annotation. Instead of a separate dedicated contextual annotation text [38], there will be video audio summarization text, which is a machine-generated piece of text that summarizes the audio in the video. Next, in the video annotation element field there will be a comment for the video, and in the video annotation element field there will also be a video annotation symbol, defining the semantic relationship between the annotation text and the video comment, the url will not be present for blocks of video annotation elements, but will be present for inline blocks video annotation elements. In order for a web browser to display these blocks of video annotation elements, the video annotation display package must contain a tagged thumbnail image of a video frame at some selected point in time with a video annotation displayed on top of it with objects such as the video annotation subject bounding box [322], a video annotation comment in a box [324] and video annotation arrow [323], modified and overlaid properly. In case the user clicks the play button, the video annotation display package should contain a video thumbnail with a video annotation overlaid. In addition, the video annotation display package must contain video audio summary text, a video annotation symbol, a video annotation comment, a username, and reputation information. Additionally, users will be allowed to modify machine-generated aspects of video annotations.

In an exemplary embodiment of the invention, the embedded video annotation element block would include a separate context variant annotation control panel [41] and a separate context reputation selector [42], as would the annotation element block [36], as shown in the exemplary embodiment of the invention in FIG. 9. Instead of a screenshot with marked text [37], the embedded video annotation item window will contain a thumbnail of the embedded video frame at some selected point in time along with the embedded video annotation displayed on top of the thumbnail of the embedded video frame at the same selected point in time. Additionally, the play button may allow the thumbnail video to be played, showing a portion of the thumbnail video and an annotation of the video. Instead of a separate dedicated contextual annotation text [38], there will be an embedded video audio summary text, which is a machine-generated piece of text that summarizes the audio in the video. Next, the embedded video annotation element field will have an embedded video annotation comment, and the embedded video annotation element field will also have an embedded video annotation symbol defining the semantic relationship between the embedded audio annotation text and the embedded video annotation comment, the url of the web page in which the embedded video will be shown in the annotation element field of the embedded video. In order for a web browser to display these blocks of embedded video annotation elements, the embedded video annotation display bundle must contain a tagged thumbnail image of a frame of the embedded video at some selected point in time with the embedded video annotation displayed on top of it, with objects such as the bounding box of the embedded video annotation object [389], comment Embedded Video Annotation Box [391] and Embedded Video Annotation Arrow [390] resized and overlaid properly. In case the user clicks the play button, the embedded video annotation display set should contain a thumbnail video with the embedded video annotation overlaid. In addition, the display set of the embedded video annotation must contain the text of a short

- 37 047005 descriptions of embedded audio, embedded video annotation symbol, embedded video annotation comment, username and reputation information. Additionally, users will be allowed to modify machine-generated aspects of video annotations.

In an exemplary embodiment of the invention, the audio annotation element block would include a separate context variant annotation control panel [41] and a separate context reputation selector [42], as would the annotation element block [36], as shown in the exemplary embodiment of the invention in FIG. 9. Instead of a screenshot with marked text [37], the sound recording annotation element window will contain a thumbnail representing the sound recording, called the sound recording's associated image thumbnail. To do this, a summary text of the audio recording will first be created, which will convert the audio recording into a text representation. This will be done using voice recognition and other audio understanding technologies. Once a text representation is created, the image associated with that audio summarization text can be found using artificial intelligence or deep learning models. This image will be called the associated sound recording image. A thumbnail representation of an associated sound recording image is a thumbnail representation of an associated sound recording image. Instead of a separate contextual annotation, the highlighted text [38] will be a summary text of the audio recording. Next, in the field of the sound recording annotation element there will be a comment of the sound recording annotation, and in the field of the sound recording annotation element there will also be a symbol of the sound recording annotation, defining the semantic relationship between the text of the sound recording summary and the sound recording annotation comment, the url will not be present for blocks of sound recording annotation elements, but it will be present for inline audio recording annotation element blocks. In order for a web browser to display these blocks of audio annotation elements, the audio annotation display bundle must contain a thumbnail of the associated audio image, short description text for the audio, a comment for the audio annotation, a symbol for the audio annotation, a user name, and reputation information. Additionally, users will be allowed to modify machine-generated aspects of audio recording annotations.

In one exemplary embodiment of the invention, the embedded audio annotation element block would include a separate context variant annotation control panel [41] and a separate contextual reputation selector [42], as would the annotation element block [36]. Instead of a screenshot with tagged text [37], the embedded audio annotation item window will contain a thumbnail representing the embedded audio, called the embedded audio embedded image thumbnail. To do this, you will first create a summary text of the embedded audio that converts the embedded audio into a text representation. This is done using voice recognition and other audio understanding technologies. Once a text representation is created, the image associated with that embedded audio summary text can be found using artificial intelligence or deep learning models. This image will be referred to as the associated audio embedded image. A thumbnail representation of an associated audio embedded image is a thumbnail representation of an associated audio embedded image. Instead of a separate contextual annotation highlighted text [38], there will be a text that briefly describes the embedded audio recording. Next, the embedded audio annotation element field will contain the embedded audio annotation comment, and the embedded audio annotation element field will contain the embedded audio annotation symbol, defining the semantic relationship between the embedded audio summary text and the embedded audio annotation comment, the url of the website where the embedded audio is found. , will be present in the annotation element blocks of the embedded audio recording. In order for a web browser to display these audio embedded annotation element blocks, the audio embedded annotation display bundle must contain a thumbnail associated with the audio embedded image, audio embedded summary text, audio embedded annotation comment, audio embedded annotation symbol, user name, and reputation information. Additionally, users will be allowed to modify machine-generated aspects of audio recording annotations.

In an exemplary embodiment of the invention, the annotation group element window would include a separate context variant annotation control panel [41] and a separate contextual reputation selector [42], as would the annotation element window [36], as shown in the exemplary embodiment of the invention in FIG. 9. Instead of a screenshot of the highlighted text [37], the annotation group element field will contain a thumbnail representing the annotation group. For image annotations, this means a thumbnail view of the image with all the image annotations included in the annotation group. Image annotations and annotation groups include embedded image annotations. For video annotations, this means a thumbnail view of a video at a specific point in time with all the video annotations included in the annotation group at that same point in time. Additionally, video thumbnails include video thumbnails with all video annotations included in the annotation group that appear throughout the entire video thumbnail. Video annotations and annotation groups include video annotations of embedded videos. For sound recording annotations, this means a thumbnail image representing the sound for all ranges of playback of the sound recording

- 38 047005 for all sound recording annotations in an annotation group. Such an image will be based on a summary text of the sound recording for all ranges of sound recording playback. In place of the selected text of an individual context annotation [38] there will be a list of annotations in this annotation group. In place of a separate contextual comment [40] there will be a user comment about this group of annotations. Additionally, these elements can be modified by users if the default machine-generated images and text are not suitable. Alternatively, the list of annotations in an annotation group may be replaced with more appropriate human-generated summary text; the url will not be present for annotation group element fields that do not have inline annotations, but the url will be present for annotation group element fields that include inline annotations. The annotation group inline annotation symbol can be used to indicate a semantic relationship between the source material and the annotations in the annotation group. The annotation group element field also specifies the number of annotations in the annotation group. In order for a web browser to display these annotation group element fields, the annotation group's display link must include a corresponding thumbnail or video thumbnail, a list of annotation object references, a corresponding summary text, a corresponding annotation comment, a corresponding annotation group annotation symbol, and the username. and reputation information. Additionally, users will be allowed to modify machine-generated aspects of annotation groups.

In one exemplary embodiment of the invention, a block of derived annotation group elements would include a separate context variant annotation control panel [41] and a separate context reputation selector [42], as would a block of annotation elements [36], as shown in the exemplary embodiment of the invention in FIG. . 9. Instead of a screenshot of the highlighted text [37], the derived annotation group item field will contain a thumbnail representing the derived annotation group. For image annotations, this means a thumbnail view of the image with all the image annotations included in the derived annotation group. Image annotations and derived annotation groups include embedded image annotations. For video annotations, this means a thumbnail image of the video at a specific point in time with all the video annotations included in the derived annotation group at that same point in time. Additionally, video thumbnails will include video thumbnails with all video annotations in the derived annotation group displayed throughout the video thumbnail. Video annotations and derived annotation groups include video annotations of embedded m videos. For audio annotations, this means a thumbnail image representing the audio for all audio playback ranges for all audio annotations in the derived annotation group. Such an image will be based on a summary text of the sound recording for all ranges of sound recording playback. In place of the selected text of an individual context annotation [38] there will be a list of annotations in this derived annotation group. In place of a separate contextual comment [40] there will be a user comment about this derived group of annotations. Additionally, these elements can be modified by users if the default machine-generated images and text are not suitable. Alternatively, the list of annotations in a derived annotation group may be replaced with more appropriate human-generated summary text, the url will not be present for derived annotation group element fields that do not have inline annotations, but the url will be present for derived annotation group element fields that include inline annotations . The annotation symbol of a derived annotation group can be used to indicate a semantic relationship between the source material and the annotations in the annotation group. In order for a web browser to display these blocks of annotation group elements, the annotation group display package must include a corresponding thumbnail image or thumbnail video, a list of annotation object links, corresponding summary text, a corresponding derived annotation comment, a corresponding derived annotation group annotation symbol, a name user and reputation information. Additionally, users will be allowed to modify machine-generated aspects of derived annotation groups.

At this point, annotation groups and derived annotation groups may appear similar in the context of the exemplary embodiments described. However, derived annotation groups, derived annotation group element fields, and derived annotation group display sets, when used, will be treated differently than annotation groups, annotation group element fields, and annotation group display sets. Annotation groups indicate a closer relationship between annotations, and derived annotation groups can be included within annotation groups. The impact of these differences can be seen by examining FIG. 44 and fig. 45.

In an exemplary embodiment, an annotation group with derived annotations would include a separate context variant annotation control panel [41] and a separate context reputation selector [42], as would an annotation group [36], as shown in the exemplary embodiment in FIG. 9. Block of annotation group elements with derived annotations and corresponding display set

- 39 047005 An annotation group with derived annotations is similar to an annotation group element block and an annotation group display bundle, except that for each derived annotation group there is information about the derived annotations, such as its constituent annotations and a comment for the derived annotation. Having this information allows derived annotations to be displayed, as shown in FIG. 44 and fig. 45. Additionally, the Annotation Group with Derived Annotation element field displays the number of annotations and the number of derived annotation groups within the derived annotation group. All this information is reflected in the annotation group display beam with the derived annotations. An annotation group element window with derived annotations will include a separate contextual variant annotation control panel [41] and a separate contextual reputation selector [42], as does the annotation element window [36]. Instead of a screenshot of the highlighted text [37], the Derived Annotation Group Item field will contain a thumbnail representing the Derived Annotation Group item. For picture annotations, this means a thumbnail image of the picture with all the annotations included in the annotation group with derived annotations. Because derived annotation groups modify images by virtue of the annotations that are contained in the derived annotation group, the set of all the annotations contained in the annotation group with the derived annotations is important. Image annotations and annotation groups include embedded image annotations. For video annotations, this means a thumbnail image of a video at a specific point in time with all the video annotations included in the annotation group with derived annotations at that same point in time. Additionally, video thumbnails will include video thumbnails with all video annotations included in an annotation group with derived annotations displayed throughout the video thumbnail. Again, what is important is the set of all annotations contained in an annotation group with derived annotations. Video annotations and annotation groups include annotations from embedded videos. For audio annotations, this means a thumbnail image representing the audio for all audio playback ranges for all audio annotations in an annotation group with derived annotations. This image will be based on a summary of the audio text for all audio playback ranges in the set of all annotations in the annotation group with derived annotations. In place of the selected text of an individual contextual annotation [38] there will be a list of annotations in that particular annotation group with derived annotations. Instead of a separate contextual comment [40], there will be a user comment about a given annotation group with derived annotations. Additionally, these elements can be modified by users if the default machine-generated images and text are not suitable. Alternatively, the list of annotations in a derived annotation group may be replaced with more appropriate human-generated summary text, the url will not be present for annotation group element blocks with derived annotations that do not have annotations embedded, but the url will be present for annotation group element blocks with derived annotations that include embedded annotations. The embedded derived annotation group symbol can be used to indicate the semantic relationship between source material and annotations in an annotation group with derived annotations. The Annotation Group with Derived Annotation element field will also indicate the number of annotations in the annotation group with derived annotations, as well as the number of derived annotation groups. In order for a web browser to display these annotation group element blocks with derived annotations, the annotation group display link with derived annotations must include a corresponding thumbnail or video thumbnail, a list of annotation object references, corresponding summary text, a corresponding annotation comment, a corresponding symbol annotation group with derived annotations, username, and reputation information. Additionally, users will be allowed to modify machine-generated aspects of annotation groups with derived annotations.

After discussing the various types of item sets, display sets, and item blocks in the context of the exemplary embodiment, changes that need to be made to FIG. 14, fig. 15, fig. 16, fig. 17 and fig. 18. These changes adapt these sequence diagrams and flowcharts to implement methods and systems associated with the features and capabilities shown and described in the written descriptions and exemplary embodiments depicted in FIGS. 35, fig. 36, fig. 37, fig. 38, fig. 39, fig. 40, fig. 41, fig. 42, fig. 43, fig. 44, fig. 45, fig. 46, fig. 47, fig. 48, and fig. 49. Sequence diagram of FIG. 14 is applied with one of the sets of data elements described previously, together with a corresponding display set and a corresponding display set, when the annotation creation process is different for the different types of annotations described previously when different types of annotations are implemented. The display set occurs where in FIG. 14, an annotation display set is used, and the display set occurs where in FIG. 14 uses a set of displayed annotations. The sequence diagram in FIG. 15 also applies to the previously described display beams and display beams. These mapping sets occur where in FIG. 15 annotation display sets arise, and

- 40 047005 displayed sets appear where in FIG. 15 uses mapped annotation sets. A display link has data for displaying either a corresponding block of elements or a corresponding display link. Depending on the exemplary embodiment, the user interface may display either a block of elements or a display bundle. Typically, a block of elements is a display bundle along with a separate contextual variant annotation control panel [41] and a separate contextual reputation selector [42]. A separate context variant annotation control panel [41] and a separate context reputation selector [42] are shown for an annotation element block [36] in the example implementation shown in FIG. 9. Sequence diagram of FIG. 16 is applied with annotation symbol display beams replaced by corresponding embedded annotation display beams, and closed annotation symbol display beams replaced by visible elements of the corresponding embedded annotation. Inline annotation display sets and inline annotation visibility sets have been discussed previously. fig. 17 and fig. 18 are applied as is, with the data element set and display sets selected from the types that were specified earlier in this document.

In one exemplary design embodiment, a basket of items [44] within an annotation organization context area [43] in FIG. 10 may include annotation element blocks, embedded image annotation element blocks, embedded video annotation element blocks, embedded audio annotation element blocks, annotation group element blocks, derived annotation group element blocks, annotation group element blocks with derived annotations, and scratchpads. Notebooks can also be displayed as a field of notebook items. All of these element block types have a separate annotation control panel [41], as shown for the annotation element block [36]. While other elements of the annotation element block [36] change for other types of element blocks, the individual context-variant annotation control panel [41] and the displayed form factor of the element block do not change. This allows these new types of item blocks to be integrated into an item basket [44]. All of these item types are created on the external site, and they appear in the item cart [44] shortly after the user creates these items. This is facilitated by the use of web sockets, which transmit information from the annotation server [61] to all connected web browsers, which must receive display packets based on access control and usage scenario. Blocks of items may be moved from the item bin [44] to a writable collection represented by a display writable collection [50], as shown in the exemplary embodiment of FIG. 11. Additionally, annotation inline blocks, image annotation inline blocks, video annotation inline blocks, and audio inline blocks can be added to the active notebook [159] using the Add to Notebook button [158] located on the contextual control panel annotations [34], as shown in the exemplary embodiment of FIG. 23. The Add to Notepad icon button [158] does not appear in the contextual option annotation control panel [34] indicated in all figures. However, such a button could be added to the contextual option annotation control panel [34] as part of an exemplary implementation. Other types of items can also be added to the active notebook [159] using a trigger to perform add to notebook. In all cases, clicking an element field in the active notebook [159] should open the corresponding element in a browser tab, scroll the page to the annotation location, and display the annotation. All different types of item fields can appear in collections. Although in the collection shown in the exemplary embodiment depicted in FIG. 31, only cells with annotations are displayed; all types of cells can be present in the collection. All comments about the folded set of annotation display elements apply to folded sets of elements of various types. Additionally, comments about window functions apply. Stacked sets of elements can be grouped by url or based on the image element, video element, audio element, or document element that is being annotated. Additionally, the types of items included in a collection do not change other properties of the collections. The collections represented by the display collections may be displayed on pages like the one shown in the exemplary embodiment of FIG. 32.

As mentioned previously, filters such as the search filter shown in the exemplary embodiment of FIG. 10 are used to receive input from users. In the case of the search filter [44] of the item basket [46], this is the search phrase. These inputs are converted into parameter values sent to the annotation server [61]. These parameter values are used by the annotation server [61] to input parameters into a query that is used by the electronic database [62]. The request includes parameter values and page number. A query with parameter values is used to determine a set of elements and order those elements sequentially. These elements include web page annotations, inline image annotations, inline video annotations, inline audio annotations, image annotations, video annotations, audio annotations, annotation groups, derived annotation groups, annotation groups with derived annotations, pdf document annotations, pdf documents. pdf format, images, videos, links and regular comments. Filters can also restrict certain types of elements. Page number

- 41 047005 points to a specific page. Additionally, the request request may include a specified number of items that must be sent to the web browser running on the client electronic device. This request contains parameters for filter values, such as search phrases and a list of items to exclude from the specified list of items, and the sequential ordering of that list of items. As discussed previously, such sequential ordering provides paging information in the exemplary embodiments depicted in FIG. 10 and fig. 31. Collection filters and collection paging operate in the same manner as previously described for the exemplary implementations depicted in FIG. 10 and fig. 32. The same applies to the pinning functionality.

The incoming messages described in the exemplary embodiment depicted in FIG. 33 will function similarly when collections include different types of items. The changes will be that users can now discuss details of more item types in collection posts. Moreover, functionality can be built in message chains specialized for each type of item. You can then implement functionality to modify the objects that are components of the annotation, approve changes to the annotation, or fork the annotation that is added to the current collection. If the discussion is centered around an image annotation, you can use selectors for each object in the image annotation to facilitate discussion. Additionally, selectors for other types of elements, such as video or audio annotations, can be built into the message thread functionality. Such functionality can transform a video annotation object bounding box [322], a video annotation comment box [324], or a video annotation arrow [323], as well as change the motion path of a video annotation bounding box [332], the motion path of a video annotation arrow [333], and the motion path of a field. Entering a video annotation comment [334]. After proposing changes in a message thread, the owner of an item can approve the changes, or the administrator can approve forking the item into a new item in the context of the current collection.

Exemplary embodiments depicted in FIGS. 28 and fig. 29 are also applicable when multiple element types are present. In this situation, the annotation server [61] and electronic database [62] will handle these different types of elements. In an exemplary embodiment, an annotation server [61] might have a REST API to handle different types of elements, and an electronic database [62] might have a schema that handles different data elements in different types of packages. This affects hardware requirements as follows. If video [319] is not stored locally and is referenced via url, having video [319] will not increase the storage requirements as much as if video [319] were stored locally. If it is stored locally or via a content delivery network [63], video [319] will have a greater impact on storage requirements. The availability of audio playback [348] may create the need to use deep learning models to handle voice and audio recognition. The presence of video [319] will increase the system's processing requirements, since video [319] typically has higher processing requirements than other types of media. To provide additional annotation capabilities to the user, in an exemplary embodiment, machine learning and deep learning models can be deployed and configured through an annotation-enabled web browser extension. You can also deploy and configure directly through an annotation-capable web browser if the annotation-capable web browser is not built using a web browser extension. This functionality can be made available to the user through a panel located on the web page or a floating toolbar. This functionality can also be accessed through menu options or the browser's fixed toolbar. A floating toolbar can be attached to a page highlight or placed in the browser sidebar. For images and inline images, this panel or floating toolbar can make available buttons for tasks such as recognizing objects in images, recognizing text in images, setting the recognition characteristics of object and text detectors, and setting the characteristics of annotation elements. The elements of an annotation are the annotation object bounding box, the annotation arrow, and the annotation comment box. The object recognition button draws bounding boxes with dotted lines around potential annotation objects. The user can then select the annotation object of interest and write a comment to the annotation in the annotation comment input field that appears. Depending on the configuration settings, the annotation arrow may be created automatically. The user may also be given the option to select multiple annotation object bounding boxes, which may result in one annotation arrow being created for each annotation object bounding box, with one end of each annotation arrow pointing towards the annotation object bounding box and the other end pointing towards the annotation input field. comment or annotation comment field, depending on whether an annotation comment was written. The placement of arrowheads is a configuration property. Another configurable property is the visibility of the annotation object's bounding box, annotation arrow, and

- 42 047005 comment fields for the annotation. Visibility characteristics include visibility, mouseover, and invisibility. Visible - for always visible annotation elements. Mouseover - for annotation elements that become visible when hovering the mouse over target areas. And invisible - for annotation elements that are always invisible. The mouseover target regions may be at least one of an annotation object or an annotation element that is visible. In an alternative embodiment, annotation object outlines are used, enclosing an enclosed region including the annotation object pixels. These annotation object outlines can be used in contrast to the annotation object's bounding boxes. Additionally, the outline of an object can be a mouseover element rather than a visible element, and this visibility characteristic can be controlled by a configuration parameter. It is also possible to have a configuration option that allows the user to choose whether to use annotation object bounding boxes or annotation object outlines. Determining annotation feature boundaries or annotation feature outlines relies on machine learning or deep learning models that can recognize objects and determine annotation feature boundaries or annotation feature outlines. Deep learning models can be run in a web browser running on the client electronic device, or using the CPU or GPU of the client electronic device. The GPU can be used using the WebGL interface provided by modern web browsers, which in turn uses the GPU of the client electronic device. Annotation feature boundaries and annotation feature contours can be determined using annotation feature detectors using region-based convolutional neural networks (R-CNNs) or object segmentation using region-based convolutional neural networks (R-CNNs). In addition to R-CNN-based models, other machine learning or deep learning techniques may be used in alternative embodiments. Since different deep learning models have different recognition characteristics, the user can input the desired recognition characteristics of object and text detectors. Based on these characteristics, different machine learning or deep learning models can be selected. Text in images can be recognized using convolutional neural networks (CNNs) designed to recognize various letters, numbers and other characters. Additionally, in the configuration options, the user can choose whether to group individual characters into words and whether to group individual words into n-grams. In addition, the user can enter words or tags through the configuration options available for recognition characteristics to limit object recognition to the specified object types that the user has entered. To work with video and embedded videos, an additional button is provided to configure the trajectories of movement of annotation elements. The techniques described for images can also be implemented in individual video frames. Additionally, deep learning models can track annotation objects over time and ensure that the bounding boxes and outlines of annotation objects move with the annotation object. This can be done by analyzing individual frames at discrete points and then using interpolation techniques between the discrete points. The result is motion paths for the annotation feature's bounding box and the annotation feature's outline. The offsets of the annotation object's contour contours are also calculated. The configuration of these motion paths and the evolution of the annotation outline is available to the user through an additional panel button, menu option, browser toolbar, or floating toolbar that handles annotation element motion path configuration options. Machine learning and deep learning models for recognizing voice and audio in video for annotation purposes can be handled using another configuration option in the panel, menu options, browser toolbar, or floating toolbar.

As used herein, the word exemplary means serving as an example, example, or illustration. Any embodiment described herein as exemplary should not necessarily be construed as preferable or superior to other embodiments, and it should not be construed as a model to be literally copied. Thus, this description should be considered as illustrative and not restrictive with respect to the present invention.

Conventional language such as, but not limited to, may, could, might, or may, unless specifically stated otherwise or otherwise understood in context as used, are generally intended to convey that some embodiments include, and other embodiments do not include, certain attributes, elements and/or steps. Thus, such conditional statements do not generally imply that the features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for making a decision, with or without user input or hints as to whether these features, elements and/or steps are included or must be performed in any particular embodiment. The headings used herein are for the convenience of the reader and are not intended to limit the scope of the invention or the claims.

- 43 047005

As used herein, the phrase referring to at least one of a list of elements refers to any combination of those elements, including individual members. By way of example, at least one of A, B or C is intended to cover A, B, C, A and B, A and C, B and C, and A, B and C. Subjunctive expressions, such as the phrase at least at least one of X, Y and Z, unless specifically stated otherwise, are understood in context to be used generally to convey that an item, term, etc. may be at least one of X, Y or Z. Thus, such conjunctive language does not generally imply that certain embodiments of the invention require at least one of X, at least one of Y and Z, at least one of Y and at least one of Z are each present separately.

The flowcharts, graphical user interfaces, sequences, systems, components and swimming activity diagrams in the figures illustrate the use case, functionality, architecture and operation of possible implementations of methods, computer program products and systems in accordance with various embodiments of the present invention. In this regard, each element in the flowchart, graphical user interface, sequence, system, component, and activity diagrams may represent a module, segment, or part of an instruction that includes one or more executable instructions to implement specified logical function(s). In some alternative implementations, the functions specified in the clause may not occur in the order shown in the figures. For example, two elements shown sequentially may, in fact, be executed substantially simultaneously, or elements may sometimes be executed in reverse order depending on the functionality involved. It should also be noted that each element that does not involve one or more users, a flowchart, a graphical user interface, a sequence, a system, a component, and an activity diagram, as well as a combination of elements, where a combination of elements does not involve one or more users, is a block. circuit, graphical user interface, sequence, system, component, and activity circuit may be implemented by special-purpose hardware systems that perform specified functions or actions or execute combinations of special-purpose hardware and computer instructions.

Aspects of the present invention are described herein with reference to flowcharts, graphical user interfaces, sequences, systems, components and operational diagrams of floating track methods, computer program products and systems in accordance with embodiments of the invention. It should be understood that each element not involving one or more users, flowchart, graphical user interface, sequence, system, component and activity diagram of a floating track, and combinations of elements not involving one or more users, in a flowchart, graphical The user interface, sequence, system, component, and activity diagram of a floating track can be implemented using computer-readable program instructions. These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device to create a machine such that the instructions, which are executed by the processor of the computer or other programmable data processing device, create means for implementing functions/actions , not involving one or more users specified in the flowchart, graphical user interface, sequence, system, component and activity diagram of the floating track, element or elements, where the element and elements do not involve one or more users.

These computer-readable program instructions may also be stored on a computer-readable storage medium that can direct a computer, programmable data processing device, and other devices to operate in a particular manner, such that the computer-readable storage medium with the instructions stored therein includes an article of article including instructions implementing aspects of the function/operation , which do not involve one or more users, specified in the flowchart, graphical user interface, sequence, system, component and activity diagram of the floating track, an item or items where the item and items do not involve one or more users.

Such code, as will be appreciated by those skilled in the art, may be stored or arranged to be stored on one or more tangible computer-readable media, such as memory chips, local or remote hard drives, optical disks, or other media that can be accessed by a processor system. to execute stored code. It should be noted that the tangible medium may be paper or other suitable medium on which the instructions are printed. For example, instructions may be obtained electronically by optically scanning paper or other media, then optionally compiled, interpreted, or otherwise processed as appropriate, and then stored in computer memory. Additionally, it should be noted that physical media may be version control stores using memory chips, local or remote hard drives, optical disks, or other media for storage. For example, instructions may be semi

-

Claims (20)

sourced from a version control repository, then compiled, interpreted or otherwise processed in a suitable manner, and then stored in computer memory. Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device to cause a series of operational steps to be executed on the computer, other programmable device, or other device to create a computer-implemented process such that the instructions executed on computer, other programmable device, or other device, implement the functions/actions specified in the flowchart, graphical interface, sequence, system, component, and floating track activity diagram element or elements. In addition, although the methods, non-transitory computer-readable media, systems and devices described herein are subject to various modifications and alternative forms, specific examples thereof are shown in the drawings and described in detail herein. However, it should be understood that the invention is not limited to the specific forms or methods disclosed, but rather, the invention covers all modifications, equivalents and alternatives falling within the spirit and scope of the various embodiments described and the appended claims. Moreover, the disclosure herein of any specific feature, aspect, method, property, characteristic, quality, attribute, element or the like in connection with an implementation or embodiment may be used in all other implementations or embodiments set forth herein. Any methods disclosed herein are not necessarily performed in the order stated. The methods disclosed herein may include certain actions taken by a practitioner; however, the methods may also include any indication by third parties of these actions, expressly or impliedly. The ranges disclosed herein also include any and all overlaps, subranges, and combinations thereof. Expressions such as at least, at least, etc. include the specified number. Numbers preceded by a term such as about or approximately include those numbers and should be interpreted as appropriate (e.g., as accurately as is reasonably possible under the circumstances, e.g. -5%, 5%, 10%, 15% and etc.). For example, about 3.5mm includes 3.5mm. Phrases preceded by such a term substantially include the phrase cited and are to be interpreted in light of the circumstances (e.g., to the extent reasonably practicable under the circumstances). For example, essentially permanent includes permanent. The detailed description is provided to enable one skilled in the art to make and use the invention and is provided in the context of the requirements for obtaining a patent. Various modifications to the exemplary embodiments will be readily apparent to those skilled in the art, and the general principles of the present invention may be applied to other embodiments, and some features of the present invention may be used without corresponding use of other features. Accordingly, the present invention is not limited to the embodiments shown, but has the broadest scope consistent with the principles and features described herein. CLAIM 1. A non-transitory computer-readable medium on which data is stored, which is software executable by a web browser application running on at least one processor, wherein the non-transitory computer-readable medium includes instructions for displaying a plurality of displayable annotation symbols embedded in the text of a web document , wherein the displayed annotation symbol has a state selected from a group of states, and wherein the group of states includes closed and open; wherein the displayed annotation symbol in the closed state displays the annotation symbol in the web document with minimal visual change to the web document and the displayed annotation symbol in the open state displays screen elements including annotative highlighting of a selected piece of text from the document, wherein the selected piece of text represents an annotation; an annotation comment and an annotation display symbol indicating the creator's selection of the annotation from a group of annotation semantic relationships, wherein the group of annotation semantic relationships includes a disagreement relationship indicating disagreement between the selected text fragment; and annotation comment; and a question relation, meaning that the annotation comment asks a question about the selected fragment - 45 047005 those texts. 2. A non-transitory machine-readable medium according to claim 1, in which the group of annotation semantic relations further includes an agreement relation, meaning agreement between the selected text fragment and the annotation comment; and an exclamation relation, meaning that the annotation comment indicates an exclamation note that represents a warning or interesting fact about the selected piece of text. 3. The non-transitory machine-readable medium of claim 1, wherein the selected portion of text is highlighted with a color previously assigned to the displayed annotation symbol, depending on the annotation semantic relationship to which the displayed annotation symbol corresponds. 4. The non-transitory computer-readable medium of claim 1, wherein the displayed annotation symbol, in the open state, displays an annotation comment field including the annotation comment; a contextual annotation options control panel including controls for a user to manage a privacy setting selected from a group of privacy states; and a background color of the annotation comment field that signifies the privacy setting, with each privacy state in the privacy state group being assigned a different background color. 5. The non-transitory computer-readable medium of claim 1, further comprising instructions for displaying a character beacon, wherein the character beacon has a state selected from a group of states including triggered and untriggered, wherein the character beacon comprises a trigger mechanism that transitions the character beacon from the untriggered state. to the triggered state and the symbol beacon, while in the triggered state, causes one displayable annotation symbol beacon to be displayed for each displayable annotation symbol, making all displayable annotation symbols more visible within the user's viewable area of the annotation-enabled web browser. 6. The non-transitory computer-readable medium of claim 5, wherein the displayed annotation symbol beacon and the corresponding displayed annotation symbol are assigned the same pre-assigned color depending on the semantic relationship of the annotation to which the displayed annotation symbol corresponds. 7. The non-transitory computer-readable medium of claim 5, wherein the trigger mechanism is a button displayed in a viewable area of a web browser with annotation support. 8. The non-transitory computer-readable medium of claim 5, wherein the trigger mechanism is a predetermined combination of input data selected from the group consisting of keyboard keystrokes and mouse button presses. 9. The non-transitory computer-readable medium of claim 6, wherein the trigger mechanism is a menu option in an annotation-enabled web browser. 10. The non-transitory machine-readable medium of claim 1, wherein the group of semantic relations of the annotation further includes an advertising relation meaning that the annotation comment is an advertisement related to the selected text fragment, and wherein the creator of the annotation is an advertiser, and the advertising display symbol of the annotation is a displayable annotation symbol corresponding to a relation indicating that the annotation comment is an advertisement related to the selected piece of text. 11. A system for supporting the connection of annotations to documents in web browsers, including a client electronic device with at least one processor and memory; a web browser and a web browser extension configured to create a display of a modified graphical user interface (GUI) of the web browser, wherein the modified GUI display includes a plurality of display annotation symbols received from the annotation server embedded in the text of the web document, wherein the displayed annotation symbol has a state selected from a group of states, wherein the group of states includes closed and open; wherein the annotation symbol display in the closed state displays the annotation symbol in the web document with minimal visual change to the web document, and wherein the annotation symbol display in the open state displays elements received from the annotation server, including highlighting the annotation of the selected piece of text from a document, with the selected text fragment representing an annotation; an annotation comment and a display annotation symbol indicating the creator's selection of an annotation from a group of annotation semantic relations, wherein the group of annotation semantic relations includes a disagreement relation indicating a disagreement between the selected text fragment and the annotation comment; and a question relation, meaning that the annotation comment asks a question about the selected piece of text. 12. The system according to claim 11, in which the group of annotation semantic relations further includes an agreement relation, meaning agreement between the selected text fragment and the annotation comment; and an exclamation relation, meaning that the annotation comment indicates an exclamation note that represents a warning or interesting fact about the selected piece of text. 13. The system of claim 11, wherein the selected text fragment is highlighted with a color previously assigned to the displayed annotation symbol, depending on the annotation semantic relationship to which the displayed annotation symbol corresponds. 14. The system of claim 11, wherein the displayed annotation symbol in the open state displays an annotation comment field including an annotation comment; a control panel providing the user with controls to manage a privacy setting selected from a group of privacy states; the background color of the annotation comment field, which signifies the privacy setting, with each privacy state in the privacy state group assigned a different background color. 15. The system of claim 11, wherein the modified GUI display further includes a character beacon, wherein the character beacon has a state selected from the group of states including triggered and untriggered; wherein the character beacon comprises a triggering mechanism that transitions the character beacon from an untriggered state to a triggered state, and the character beacon, while in the triggered state, causes the display of screen elements including the displayed annotation symbol beacon, making the location of the displayed annotation symbol more prominent for each displayed symbol annotation displayed in the user-visible area of an annotation-enabled web browser. 16. The system of claim 15, wherein the displayed annotation symbol beacon and the corresponding displayed annotation symbol are assigned the same pre-assigned color depending on the semantic relationship of the annotation to which the displayed annotation symbol corresponds. 17. The system of claim 15, wherein the trigger mechanism is a button displayed in a viewable area of a web browser with annotation support. 18. The system of claim 15, wherein the triggering mechanism is a predetermined combination of input data selected from the group consisting of keystrokes on a keyboard; and keystrokes on the keyboard; and pressing buttons on the mouse. 19. The system of claim 15, wherein the trigger mechanism is a menu option in an annotation-enabled web browser. 20. The system of claim 11, wherein the annotation semantic relationship group further includes an advertising relation indicating that the annotation comment is an advertisement related to the selected text fragment, and wherein the creator of the annotation is an advertiser and the advertising display symbol of the annotation is a display symbol annotation corresponding to the relationship meaning that the annotation comment is an advertisement related to the selected text fragment, and the advertisement comes from the ad server. -