Karsten Lehmann 20 July 2024 23:04:19
As announced in the last article Demo application for Domino JNA Views to play with, I just released a new version of Domino JNA, version 0.9.53, with these changes; - fixed a VirtualViewNavigator issue where a navigator created via buildFromCategory(...) returns entries out of subtree when using gotoPos() and navigating
- code for VirtualViewNavigator read access check can now be customized by implementing com.mindoo.domino.jna.virtualviews.security.IViewEntryAccessCheck or subclassing com.mindoo.domino.jna.virtualviews.security.ViewEntryAccessCheck when creating the navigator
- new API method com.mindoo.domino.jna.virtualviews.VirtualViewNavigator.entriesForwardFromPosition(String, SelectedOnly)
These were findings and feedback I got from creating/showing my Virtual View API demo application at the recent OpenNTF developer webinar. 
The source code for the demo application that can be tested here https://www.mindoo.de/test/jnavirtualviews.nsf/sunburst.xsp is now also available on GitHub: https://github.com/klehmann/JNA-Virtual-Views-Demo Karsten Lehmann 18 July 2024 17:50:24
Today I attended the "Developer Variety Hour" webinar of OpenNTF and did a demo of the new Virtual View API of Domino JNA. I built a small web application that combines a Sunburst diagram (taken from the D3 website, enhanced by a training session with Anthropics Claude AI chat :-) ) with a categorized view, built on top of a Bootstrap 5 table. The application looks like this: 
and it's available for you to play with here: https://www.mindoo.de/test/jnavirtualviews.nsf/sunburst.xsp Both components fetch their data from an XAgent style XPage where a Virtual View is used to read document information from two fakenames databases: public Optional< VirtualView > getViewById(String id) { VirtualView view = null; if ("fakenames_bycontinent_categorized".equals(id)) { view = VirtualViewFactory.INSTANCE.createViewOnce(id, 17, // view version number 10, TimeUnit.MINUTES, // keep in memory for 10 minutes (viewId) -> { return VirtualViewFactory.createView( // use a Java function to compute the "Continent" category: new VirtualViewColumn("Continent", "$1", Category.YES, Hidden.NO, ColumnSort.ASCENDING, Total.NONE, new VirtualViewColumnValueFunction< String >(1) { @Override public String getValue(String origin, String itemName, INoteSummary columnValues) { String companyName = columnValues.getAsString("CompanyName", ""); Map lookupMap = getCompanyContinentMap(); return lookupMap.getOrDefault(companyName, "Unknown"); } }), // use formula language for the remaining columns: new VirtualViewColumn("Company Name", "$2", Category.YES, Hidden.NO, ColumnSort.ASCENDING, Total.NONE, "@Left(CompanyName;1) + \"\\\\\" + CompanyName"), new VirtualViewColumn("Lastname", "Lastname", Category.NO, Hidden.NO, ColumnSort.ASCENDING, Total.NONE, "Lastname"), new VirtualViewColumn("Firstname", "Firstname", Category.NO, Hidden.NO, ColumnSort.ASCENDING, Total.NONE, "Firstname"), new VirtualViewColumn("CompanyName", "CompanyName", Category.NO, Hidden.YES, ColumnSort.NONE, Total.NONE, "CompanyName") ) // load data from two fakenames databases .withDbSearch("fakenames1", "", "fakenames.nsf", "Form=\"Person\"") .withDbSearch("fakenames2", "", "fakenames2.nsf", "Form=\"Person\"") .build(); }); } return Optional.ofNullable(view); } Initial index generation for about 40.000 fakenames docs takes 5 seconds on our server. The view is set to expire after 10 minutes and will be automatically be rebuilr. The first column value is computed via Java code and looks up the continent for the company name of a fakenames person (let's call it a JOIN operation). Since all data access is done in-memory (no legacy Domino view is required), REST API response times are ultra-fast and take 50ms for each request. Part of those 50ms is opening the two fakenames database and checking if anything relevant has been changed/added/deleted. As presented in the webinar, changing DB documents is immediately reflected in the web app. Building this demo app this week I discovered a few issues in the Virtual View API of Domino JNA 0.9.52 especially when processing NSF changes (the code produced duplicate rows). I will publish a new Domino JNA version this week with fixes. Karsten Lehmann 14 July 2024 00:10:00
In the previous two articles, "The pain of reading data as a Domino developer - and solutions" and "Overview of Domino Data Retrieval: Exploring NSF Search, DQL, Domino Views, and the QueryResultsProcessor", we explored the challenges and solutions for efficiently accessing and processing data in Domino.
Now, let's delve into the next evolution in Domino data handling.
Version 0.9.49 of our Domino JNA open-source project introduced a new API called "Virtual Views", built with all the NIF, formula search, DQL, and QueryResultsProcessor history and limitations in mind.
In short, you define the view structure as a VirtualView object in your Java code (columns with sorting, categorization, total, and average values) and then add one or more data providers that incrementally push data into the view (or remove it).
This article will guide you through the innovative features and capabilities of the Virtual View API, demonstrating how it simplifies and enhances data retrieval and manipulation in Domino.
Now, let's take a closer look at how you can leverage the Virtual View API to define and manage views dynamically in your Java code.
To illustrate, consider the following code:
someExternalData.put("Revoco", "Revoco Street 5, Los Angeles");
someExternalData.put("Omnis", "Omnis Boulevard 12, New York");
// by using "createViewOnce", we mark the view to be stored in memory,
// as a version "1" and to auto discard it
// after 5 minute of inactivity (just for testing, in production you'd use a higher value)
// changing the version number to "2" would force a new view to be created
VirtualView view = VirtualViewFactory.INSTANCE.createViewOnce("fakenames_namelengths",
1, // version "1"
5, TimeUnit.MINUTES, // auto discard after 5 minute of inactivity
// (calling createViewOnce resets the counting)
(id) -> {
return VirtualViewFactory.createView(
new VirtualViewColumn("Lastname", "Lastname",
Category.YES, Hidden.NO, ColumnSort.ASCENDING, Total.NONE,
"Lastname"),
new VirtualViewColumn("Firstname", "Firstname",
Category.NO, Hidden.NO, ColumnSort.ASCENDING, Total.NONE,
"Firstname"),
new VirtualViewColumn("Total Name Length", "TotalNameLength",
Category.NO, Hidden.NO, ColumnSort.NONE, Total.SUM,
new VirtualViewColumnValueFunction< Integer >(1) {
// this 1 is a version number for
// the column function, might become
// relevant later when we store the index to disk
@Override
public Integer getValue(String origin, String itemName,
INoteSummary columnValues) {
return columnValues.getAsString("Firstname", "").length() + 1 +
columnValues.getAsString("Lastname", "").length();
}
}),
new VirtualViewColumn("Average Name Length", "AverageNameLength",
Category.NO, Hidden.NO, ColumnSort.NONE, Total.AVERAGE,
new VirtualViewColumnValueFunction< Integer >(1) {
@Override
public Integer getValue(String origin, String itemName,
INoteSummary columnValues) {
return columnValues.getAsString("Firstname", "").length() + 1 +
columnValues.getAsString("Lastname", "").length();
}
}),
new VirtualViewColumn("Company Address", "CompanyAddress",
Category.NO, Hidden.NO, ColumnSort.NONE, Total.NONE,
new VirtualViewColumnValueFunction< String >(1) {
@Override
public String getValue(String origin, String itemName,
INoteSummary columnValues) {
// poor man's JOIN :-)
// we fetch the company address from a map
// using the company name as key
String companyName = columnValues.getAsString("CompanyName", "");
return someExternalData.getOrDefault(companyName, "");
}
}),
new VirtualViewColumn("Last Update", "LastMod",
Category.NO, Hidden.NO, ColumnSort.NONE, Total.NONE,
"LastMod"),
// required to have the CompanyName value available in the summary
// buffer so that the Java column function can use it
new VirtualViewColumn("Company Name", "CompanyName",
Category.NO, Hidden.YES, ColumnSort.NONE, Total.NONE,
"CompanyName")
)
// add one or more data providers with an origin id (here "myfakenames1")
.withDbSearch("myfakenames1",
"Server1/Mindoo", "fakenames.nsf",
"Form=\"Person\"")
// let's move the categories above the docs like in Windows Explorer
.withCategorizationStyle(CategorizationStyle.CATEGORY_THEN_DOCUMENT)
.build();
});
In this example, column values are computed via formula and Java code, and a JOIN operation with an external Java Map is performed.
More examples can be found here: https://github.com/klehmann/domino-jna/blob/develop/domino-jna/src/test/java/com/mindoo/domino/jna/test/TestVirtualView.java
The data provider implements a simple Java interface, making it easy to add non-Domino data to the view. For Domino, we built several standard implementations that can be mixed and matched for one or multiple databases:
- datasource 1: run NSF search with a formula (incrementally), can search data and design documents, optional post processing with FT search
- datasource 2: profile documents filtered with a formula
- datasource 3: read note ids from a folder (incrementally), ideal for displaying folder content in different ways
- datasource 4: compute column values from any list of note IDs (e.g. a DQL result)
The VirtualView class manages the categorization tree and document sort order, aggregating computation results on category levels up to an imaginary root element.
Readers lists
Readers lists are a first-class citizen in the Virtual View API. For documents, we store the individual readers within the view index entry, along with the origin database information. When traversing the view, we compare the readers list with the usernames list of the current user in that specific database. This also works with ACL roles which are defined per database.
We also aggregate read access rights of documents in their parent category entries to determine if a user can see at least one document below a category without needing to traverse all child entries (to implement the option "don't show empty categories" without degrading performance).
These aggregated readers stats can be read in code, offering insights into the distribution of read access rights across multiple databases.
Virtual Views do not need to be created on a per-user basis. The VirtualView object contains all data the server is allowed to see. To traverse the virtual view, we built our own VirtualViewNavigator, instantiated for a specific user and for the whole view or a single category:
VirtualViewNavigator nav = view
.createViewNav()
.withCategories()
.withDocuments()
.withEffectiveUserName("CN=Karsten Lehmann/O=Mindoo")
.build();
// other methods to build the navigator:
// .buildFromCategory("Abbott");
// .buildFromDescendants(otherViewEntryInTheView);
nav.expandAll().collapse("1.1");
if (nav.gotoFirst()) {
do {
VirtualViewEntryData currentEntry = nav.getCurrentEntry();
String firstName = currentEntry.getAsString("firstname", "");
String lastName = currentEntry.getAsString("lastname", "");
}
while (nav.gotoNext());
}
gotoPrev() and gotoNext() manage the expanded/collapsed entries. If you prefer a non-goto syntax, methods like entriesForward(), entriesBackward() and others return Java Streams:
Stream stream = nav
.entriesForward()
.skip(3000)
.limit(200);
stream
.forEach(((entry) -> {
String firstName = entry.getAsString("firstname","");
String lastName = entry.getAsString("firstname","");
}));
The view index is currently stored in memory. We may add a feature to save indexes to disk for quick reloading after an HTTP task restart.
Conclusion
Here is the full feature list for the Virtual View API in its first version:
- Multi-DB views
- Works locally, client-server and server-server
- View structure similar to Domino (multi-level categorization, sorted columns)
- Support for sums/average values
- Compute column values via formula or Java code
- Option to place categories above or below documents in categorized views
- Incremental view updates (no rebuild required)
- Full control over when the view is updated, with optional read locks for exclusive access
- Server-populated views, shared across users
- User-specific view entry visibility checks (DB ACL level and usernames list for each DB compared with computed document readers list)
- Aggregated readers stats for categories to quickly skip empty categories for users
- Collected readers stats for analysis purposes
- Several data providers combined to produce view data
- Standard Domino data providers for data/profile/design docs
- Custom non-Domino data support
- VirtualViewNavigator for reading view entries visible to a user, with support for expanded/selected entries, upwards/downwards paging, keyword, and range lookups
- Fast performance (e.g., processes 40,000 fake name docs and builds the view in 2-3 seconds)
- VirtualView currently stored in Java heap (each VirtualViewEntry with a ConcurrentSkipListMap for sorted children), with potential future support for disk serialization
- Not Domino-specific; works for other kinds of data
I hope you enjoy this new feature, try it out, and provide feedback! Karsten Lehmann 13 July 2024 23:29:18
As you read in the previous article "The pain of reading data as a Domino developer - and solutions", looking up data on Domino is not as easy as it seems - especially compared to other platforms like SQL. Let's explore the available options.
NSF search with formula
For many years, formula language was the only universal search language on the platform. However, searching the database took time (method lotus.domino.Database.search(String formula, DateTime dt, int max)) since each document had to be scanned and processed by the formula. The result, a DocumentCollection, is said to have "no particular order," but this isn't entirely accurate. The underlying C data structure of the returned DocumentCollection is an IDTable containing note IDs. To store it efficiently, early Domino core developers decided to sort the note IDs in ascending order.
For more on the storage format of IDTables, refer to this interesting knowledge base article:
https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0026700
In data processing, the order in which documents are returned by the search call (NSFSearch in the C API) does not matter. You search the entire database and process it (one by one or quickly with one of the stamp methods). To improve performance, the NSFSearch call returns a TIMEDATE value that you can use for subsequent calls to fetch only documents created, changed, or deleted since the given time and whether they do or no longer match the selection formula.
Normal NSFSearch calls return the whole summary buffer for each matching document, which can contain a lot of irrelevant information. However, by using a special undocumented format for the compiled search formula (with merged column formulas), some undocumented flags, an undocumented item name ("$C1$"), and an undocumented NSFSearchExtended3 call, things get more interesting.
With this method, the search operation only copies the summary buffer data you are interested in, speeding up the search. The item "$C1$" contains the document's readers list, showing who can view a document. If readers are present, it also includes the document's authors. This means there's no need to manually extract this data in your own code; the NSF search operation handles it.
Our open-source project, Domino JNA, utilizes this powerful search method in the com.mindoo.domino.jna.NotesSearch class, which is an excellent tool for syncing Domino data with external systems or building custom indexes.
These undocumented flags, item and calls are the magic behind Domino views:
Domino views
End users typically want tabular data sorted by one or more columns, not data sorted by note ID. Domino views are designed for this purpose. They consist of a search formula to select relevant data and column information to visualize the data in a specific order.
Domino views provide multi-level categories for drilling down into data, which can be expanded/collapsed, and where sums and averages are aggregated (e.g., costs per team, costs per department, costs for a whole company). The view index represents a persistent and always up-to-date formula search result of one NSF database. Only one instance of a specific view index exists on the server. Since view entries contain readers information, the Notes Indexing Facility (NIF) skips rows that a user is not allowed to see by comparing their personal user names list with the allowed readers of a document.
String comparison takes time, so if you have a database with 1 million documents and you can only see 10, traversing a view is not fast because 999,990 rows have to be skipped. Domino 14 includes optimizations for these edge cases, such as maintaining an IDTable per user with document note IDs they are allowed to see. This allows the NIF code to skip string comparison and just check if the IDTable contains the note ID. However, these optimizations are not enabled by default for all users.
Domino view columns can display document item content, values computed via formula language (e.g., Lastname + ", " + Firstname), or special values like child and descendant counts read from the view index itself.
For more details on NIF, check out John Curtis's blog post:
https://jdcurtis.blog/2019/12/16/tdr-notes-indexing-facility
Folders
Folders work like Domino views, but their content (note IDs) is not retrieved via formula search. Instead, it is added manually by the end user or the application developer. The main use case of a folder is to pick and bookmark documents for future analysis or processing.
Domino Query Language
Introduced in Domino 10, the Domino Query Language (DQL) provides a concise syntax for finding documents based on a wide variety of terms. It leverages existing design elements without requiring detailed code to access them.
Like formula language, DQL can filter documents from a single NSF and return an "unsorted" IDTable. Here is an example of DQL:
Order_origin in ('Detroit', 'Albuquerque', 'San Diego') and Date_origin >= @dt('2014-07-15') and Date_origin <@dt('2015-07-14’) and
partno in all ( 389, 27883, 388388, 587992 ) and not in ('Special Processing', 'Special2' , 'Soon to be special’) and not sales_person in ('Christen Summer', 'Isaac Hart')
DQL provides a more efficient way to perform searches compared to formula language. A query planner analyzes a DQL statement and finds the best strategy to quickly reduce the number of relevant documents, such as by doing view lookups or FT searches.
A DQL search always returns the complete search result and does not support incremental searching. You can specify views, folders, or document collections to limit the results:
in ('TrudisDocs', ‘Orders’, ’Special orders folder 1’, ‘Old_orders 2’)
For more DQL examples, visit:
https://help.hcltechsw.com/dom_designer/14.0.0/basic/dql_simple_examples.html
As of Domino 14, a DQL search only covers data documents, not design documents.
For more details on DQL, see this Admincamp presentation from HCL:
https://admincamp.de/konferenz/ent2019.nsf/bc36cf8d512621e0c1256f870073e627/6c2e835120d74a18c1258327003f8d83/$FILE/T3S4-Demo%20and%20Deep%20Dive%20-%20Domino%20General%20Query%20Facility%20.pdf
QueryResultsProcessor
Introduced in Domino 12, the QueryResultsProcessor processes a list of documents, reads selected values from the documents' summary buffer, and creates tabular data with sorted, categorized, or unsorted columns. The processing result can be returned in JSON format for web applications or materialized as a Domino view in any NSF as a QRP view.
Unlike standard Domino views, data from multiple NSFs can be combined in a single QueryResultsProcessor call. For JSON output, there is no paging support (skip/limit), and the result is recomputed on every call. The QRP view is created once and does not update its content, making it suitable for producing snapshots of Domino data at a point in time.
Since you can use the normal View APIs (e.g., the ViewNavigator), returning paged data is not difficult. However, the QRP view does not store any readers lists on the row level. The view contains the data that the user creating it was allowed to see. You can only restrict access on the view level, e.g., to hide financial data from normal employees.
The primary purpose of QRP views is to produce a snapshot of Domino data at a point in time. They are more of a reporting tool than a good option for real-time queries, because you would need to build one QRP view for each user/user group and discard it as soon as the underlying data has changed, since there is no in-place index update.
John Curtis has documented the QueryResultsProcessor in a series of blog articles:
https://jdcurtis.blog/2021/11/29/the-query-results-processor-part-one/
https://jdcurtis.blog/2021/11/30/the-query-results-processor-part-two/
https://jdcurtis.blog/2021/12/02/the-queryresultsprocessor-part-three/
Curtis mentioned plans for the future of the QueryResultsProcessor, including refreshable QRP views, joins/lookups across QRP input collections, and incorporating external data into QRP views. However, he retired at the end of 2022, and the QueryResultsProcessor API has since lost momentum.
What is missing?
As described above, using NSFSearch, it is not difficult to mirror Domino data in real-time in other databases with more capable query languages like SQL, GraphQL, or Cypher (for the Graph DB fans). You can let Domino do all the work, including computing formula values and readers lists.
In the past, we built web applications for customers where we pulled the content of 30 NSFs (one for each department) into the JVM heap (max heap set to 30 GB) on HTTP task startup. We used CQEngine (https://github.com/npgall/cqengine) to build indexes for the data, resulting in extremely fast REST APIs with dynamic filter and sorting options.
However, these are custom solutions for each use case. Instead, there should be a standard way that fits many use cases and feels "Domino-like."
Let's take the QueryResultsProcessor concept to another level! Let's reinvent the wheel one more time! :-)
In the next article, I will introduce Domino JNA's new Virtual View API.
Stay tuned! Karsten Lehmann 19 June 2024 11:02:08
On my endless path of reinventing the wheel regarding Domino APIs, my latest adventure has once again led me to find efficient and powerful ways to query Domino data. For many years, this topic has been a pain point for me. Back in the IBM Design Partner program, I wrote many entries in the discussions database, asking IBM core development for better and faster ways to read Domino data. View traversal speed In the old days, according to Domino core developers, there was no priority for LotusScript or Java API methods to read data as fast as possible, "since most of the code was running at night in agents anyway" (that is what they told me). Things changed a bit when XPages were introduced in 2009. Core development began to tweak the codebase for performance, for example, to not just read one view row at a time in the ViewNavigator API, since the underlying C API call NIFReadEntries has never had this restriction. It has others; the return data buffer has a max size of 64K, but depending on the type of view data, this buffer can contain lots of rows. For example, if you just read note IDs (4 bytes each), one NIFReadEntries call returns about 16,000 rows at once. New API methods like ViewNavigator.setBufferMaxEntries(...) and ViewNavigator.setCacheGuidance(...) were the result. They sped up view traversal A LOT. Unfortunately, Domino core development thought there was no need to ever write documentation for them. Check for yourself: Go into your Domino Designer help and try to find information about these methods, same with a Google search. At least you will find a few community blogs (like ours) that wrote about this topic and how to use these methods. This video on YouTube might also help. Data consistency In addition to faster view traversal, Domino core development also started working on ways to read view data more consistently and better handle concurrent data updates. Compared to SQL databases or more modern NoSQL databases like CouchDB, Domino has always had poor data consistency. Although transaction support has existed in the Domino API since Domino 12, these transactions can only be used to modify multiple database documents atomically. They do not cover reading data from views. For example, if you want to read all view rows matching a lookup key using the publicly available C API methods, you would open the view (NIFOpenCollection), find the position of the first row matching the lookup key and determine the number of matches (NIFFindByKey, which returns a position like "1.2.3"), and start reading row data at that position (using one or more NIFReadEntries calls, depending on how much row data fits into the 64K buffer). These are three separate C API calls. The view index that you are working on is NOT your private copy. So, if other users change data at the same time and the view indexer kicks in, things become challenging. To be fair, you receive signals from NIFReadEntries (e.g., SIGNAL_ANY_CONFLICT) indicating that the view index or view design has changed in the meantime. This means that your start position "1.2.3" might be completely wrong. If you know the note ID that your code read last in addition to its position, NIFLocateNote might be handy. It tries to find the new position of this row "nearby." However, it's possible that this exact document no longer matches the lookup key and has been moved somewhere else in the view. Instead, you could re-run your lookup repeatedly until none of the NIFReadEntries calls signal a view index change. If only there were a way to prevent view index updates while lookups were running! NIFFindByKeyExtended2 to the rescue The C API of Domino 12 exposed a new method, NIFFindByKeyExtended2, which had been added to Domino years earlier. NIFFindByKeyExtended2 combines NIFFindByKey and the first NIFReadEntries call in an atomic manner. This means that the view index is locked, ensuring data consistency, at least if you don't have more than 16,000 rows for the same lookup key, or significantly fewer rows if you are also reading the column values (e.g., in one of our applications, only 70 rows filled the entire 64K buffer). I am not a C developer! Do I have to care? Life as a LotusScript or Java developer on Domino is easier than for C developers. You just call View.getAllDocumentsByKey(...) or View.getDocumentByKey(...), and the black magic happens under the covers. These methods have been using NIFFindByKeyExtended2 internally for many years, I think since R9 (or the undocumented NIFFindByKeyExtended3, which returns the result in a callback). There is a small chance that the document in the returned DocumentCollection might have changed in the meantime and no longer matches the lookup key that produced the DocumentCollection. As a paranoid person, you could check this. But as soon as the document is loaded and concurrent updates to it happen, you get a save error and can decide in your code if you want to reload the document or create a save conflict. I use View.setAutoUpdate(false), does this help to get consistent view data? In short: no. There is an old article in the Domino development wiki (https://ds-infolib.hcltechsw.com/ldd/ddwiki.nsf/dx/View.AutoUpdate_?OpenDocument&sa=true) which explains the AutoUpdate flag. It just changes the strategy how the API recovers in case of view index changes: AutoUpdate=true means "use NIFLocateNote and go on reading at the new position". AutoUpdate=false means "don't care, just go on reading at the current position" The same applies for LotusScript/Java as for C - you don't get a private copy of the view index. So things can change all the time and they sometimes shine through when using the LotusScript/Java API:
NotesException: Notes error: Entry not found in index (Articles\Latest Issue) at lotus.domino.local.ViewNavigator.NgotoEntry(Native Method) at lotus.domino.local.ViewNavigator.gotoX(Unknown Source) at lotus.domino.local.ViewNavigator.gotoChild(Unknown Source) at lotus.domino.local.ViewNavigator.getChild(Unknown Source) NotesException: Notes error: Entry not found in index (Articles\Quota Computation) at lotus.domino.local.ViewNavigator.NgotoEntry(Native Method) at lotus.domino.local.ViewNavigator.gotoX(Unknown Source) at lotus.domino.local.ViewNavigator.gotoNextSibling(Unknown Source) at lotus.domino.local.ViewNavigator.getNextSibling(Unknown Source) Here, the ViewNavigator is trying to change the cursor position and notices that the view index has changed in a way that it cannot recover. These are actual exceptions from production apps. By using setBufferMaxEntries() / setCacheGuidance(), these errors might happen less often, because the next children or sibling entries are already precached in the buffer. But the ViewNavigator precaching is only working with AutoUpdate=false and the cache is invalidated when you change navigation direction. When the ViewNavigator then runs out of cache entries, it does not care about view index changes and consistency. Hooray. What about DQL? Does that help? In short: yes. When using DQL, it is HCL's responsibility to ensure data consistency during view lookups, FT searches, or NSF scans with a search formula. The DQL engine returns matches as an IDTable for C developers (compressed storage of note IDs), which the LotusScript/Java API translates into a DocumentCollection. This translation is slower because it unnecessarily loads each document even if you only need the note IDs. If you believe you have found an example of data inconsistency, create a reproducible test case and submit a support ticket. Cliffhanger :-) In the next article, I will discuss the new QueryResultsProcessor in Domino 12, covering its capabilities and limitations. I will also explain how I rebuilt cross-database Domino view indexing in Domino JNA from scratch, avoiding these limitations. So stay tuned. Karsten Lehmann 18 November 2022 22:29:22
Once again I had to ask HCL development for setup instructions how to launch the HCL Notes Standard Client from an Eclipse IDE, this time for Notes 12.0.1 FP1. The instructions I had gotten earlier for Notes 10 have not changed much (I think there's one additional VM argument at the end of the list), but enough so that they did not work anymore.
The document links to Eclipse Neon 4.6.3, but it's still working for me in later versions, I tried it with Eclipse 2020-09 (4.17.0, 64 bit). I used my local installation directory in the text below (C:\Program Files (x86)\HCL\Notes) and the plugin version for com.ibm.rcp.base in Notes 12.0.1 FP1. Of course these need to match your local environment.
1. Download Eclipse Neon (4.6.3) or later from link below:
http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/neon/3/eclipse-committers-neon-3-win32-x86_64.zip
2. Get HCL Notes 12.0.1 FP1
I used the instructions with the 32 bit Notes Client so far.
3. Go to Windows => Preferences => Java => Installed JREs - Add => Standard VM =>
- JRE home:
C:\Program Files (x86)\HCL\Notes\jvm - JRE name: “Notes JRE”
- Select the JRE to point to this and Apply
4. Go to Windows => Preferences => Plug-in Development => Target Platform - Add => default => Target Content
- Name: “Notes Target”
4a. In Locations Tab: - Add => Directory => Location =>
C:\Program Files (x86)\HCL\Notes\framework\rcp\eclipse\plugins - Add => Directory => Location =>
C:\Program Files (x86)\HCL\Notes\framework\shared\eclipse\plugins - Finish
- Select the Target platform to point to “Notes Target” and Apply
4 b. In Content Tab: - go through the plugin list
- for duplicate plugins, deactivate the older one
- only relevant if you have installed a fixpack
4 c. In Environment Tab:
Please select following settings: - Operating System: win32
- Windowing System: win32
- Architecture: x86
- Locale: en_US - English (United States)
5. Go to Windows => Preferences => Run/Debug => String Substitution
New => Add 2 strings (change plugin version and path based on your setup)
5a. Name: rcp_home
Value: C:\Program Files (x86)\HCL\Notes\framework
5b. Name: rcp_base
Value: C:\Program Files (x86)\HCL\Notes\framework\rcp\eclipse\plugins\com.ibm.rcp.base_10.0.0.20211117-0921
OK
Close this Preferences Window
6. Put the following file inside the below plugin:
C:\Program Files (x86)\HCL\Notes\framework\rcp\eclipse\plugins\com.ibm.rcp.base_10.0.0.20211117-0921
rcp.security.properties
7. Open Debug Configurations in the ‘Debug Perspective’ => Eclipse Configuration => New
7a. In the Main tab:
Name: NotesDebug
Program to Run => Run a product => com.ibm.notes.branding.notes
Java Runtime Environment => Runtime JRE => Select “Notes JRE” i.e. the one we added in step 3
7b. In the Arguments tab:
Program Arguments:
-clean -console -debug -log -personality com.ibm.rcp.platform.personality -config notes
VM Arguments:
-Xquickstart
-Xss384k
-Xshareclasses
-Drcp_home="${rcp_home}"
-Drcp.install.config=user
-Dosgi.install.area="${rcp_home}\eclipse"
-Disa.ignoreESR=true
-Dcom.ibm.pvc.osgiagent.core.logfileloc="${rcp_home}\rcp"
-Dcom.ibm.pvc.webcontainer.port=0
-Declipse.pluginCustomization="${rcp_home}\rcp\plugin_customization.ini"
-Djava.security.properties="${rcp_base}\rcp.security.properties"
-Declipse.registry.nulltoken=true
-Djava.protocol.handler.pkgs=com.ibm.net.ssl.www.protocol
-Djava.util.logging.config.class=com.ibm.rcp.core.internal.logger.boot.LoggerConfig
-Dosgi.hook.configurators.exclude=org.eclipse.core.runtime.internal.adaptor.EclipseLogHook
-Dosgi.framework.extensions=com.ibm.rcp.core.logger.frameworkhook,com.ibm.rds,com.ibm.cds
"-Xbootclasspath/a:${rcp_base}\rcpbootcp.jar"
-Xdump:system:events=user
8. Apply => Debug – This will launch your Notes in Debug mode.
Ensure that whenever you launch the Notes from Eclipse, there should be no other instance of Notes already running. If so, please close Notes and then launch from Eclipse to debug. Karsten Lehmann 17 November 2022 23:24:41
As Jesse blogged there are some issues in the new Windows 64 Bit Notes Client 12.0.2 that break the "Import Local Update Site" functionality of the standard Update Site template of Domino. To fix this and other issues, I just uploaded a new release of the Open Eclipse Update Site on OpenNTF.
Here is the fixlist: - This release fixes LS errors in the new Windows 12.0.2 64 Bit client that occur because LS tries to load SWT Java classes, but they do not seem to be available in the Windows 64 Bit client (same for Mac/64 which had already been fixed in earlier releases).
- In addition we fixed feature deletion. You can now delete disabled features, not just enabled features like in 1.1.
- "IBM" has been replaced with "HCL" in several areas to match the current R11 standard Update Site template of Domino.
- broken links in the about/using this database pages have been fixed
Karsten Lehmann 25 October 2022 10:32:42
This is a shameless copy of Mikkel Heisterberg's blog article (https://lekkimworld.com/2010/04/14/java-class-line-numbers-for-plugin-developers/) to have this helpful information twice in the Google index and save me time when I need it in the future. :-)
Hiding line numbers in stacktraces for performance reasons is pretty high on my top list of the most stupid ideas in Notes/Domino history.
If you’ve been tasked with developing and/or debugging Java extensions for the Notes 8 client you know that line numbers has been missing from the stacktraces produced by Notes. This can be a real problem when trying to debug stuff in a production client. There has been some discussion among the ones of us developing these extensions on how to enable these line numbers. The other day this information was provided by Srinivas Rao of IBM and I wanted to publish it here for all to read.
Line numbers are removed from the classes added to the shared class cache to reduce the memory needed for the memory mapped classes. To re-enable the line numbers, one needs to edit the frameworkrcpdeployjvm.properties file and add comment out the ignorelinenumbers vm argument. However, if the classes have already been added to the JVM shared class cache, then they will have been added without line numbers. Either comment out the shared class cache (which will dramatically affect performance at startup) for temporary work, or shutdown notes and remove the shared class cache so that it can be repopulated with classes with line numbers. Of course, this will also affect the startup performance, but not so much as not having a cache
These are two of the key lines … to comment them out, add a # to the front of the line
vmarg.Xnolinenumbers=-Xnolinenumbers
vmarg.Dshare=-Xshareclasses:name=xpdplat_.jvm, **line cont**
controlDir=${prop.jvm.shareclasses.loc}, **line cont**
groupAccess,keep,singleJVM,nonfatal
jvm.shareclasses.loc=${rcp.data}/.config/org.eclipse.osgi
The shared class cache is typically located in the data/workspace/.config/org.eclipse.osgi Karsten Lehmann 13 July 2022 10:10:42
Since Domino 8.5.3, there is a built-in facility that checks if all server processes are running. When the server detects that one has disappeared/crashed, the server triggers an NSD to document this event and crashes/restarts. In general, this is a clean approach to keep the server in a consistent state. But when you develop Java applications against the Domino Server's Java API, those Java processes are treated as child processes and monitored as well It's important to develop the applications in a clean way, so every Thread accessing Domino objects should be initialized with NotesThread.sinitThread() and properly uninitialized with NotesThread.stermThread(). To ensure that this is also the case when errors occur in code, use try/finally: NotesThread.sinitThread(); Session session = null; try { session = NotesFactory.createSession(); // do something with the session object } catch (NotesException e) { e.printStackTrace(); } finally { if (session!=null) { try { session.recycle(); } catch (NotesException e) {e.printStackTrace();} } NotesThread.stermThread(); } This works in general, but often during development, you step through the code in debugger and just stop its execution, which produces this annoying result: 
Fortunately it is possible disable the automatic Domino crash/restart with this Notes.ini variable: NO_TERM_EXIT_NO_PANIC=1 The server still complains about the missing child process, but keeps on running. Of course, it would be even better if there was a way to exclude specific processes/process names from the monitoring. Karsten Lehmann 1 June 2022 00:43:40
Today we released version 0.9.48 of Domino JNA as OSGi plugin for XPages developers and on Maven Central.
Here is a list of new features: - API to read and write the Notes workspace (read, create and modify pages and icons, change page order, move replicas on top etc.)
- Formula execution now supports more than 64k of return data
- API to apply security to formula execution (e.g. prevent Notes.ini changes)
- API for QueryResultsProcessor (produces JSON and views)
- Java 8 date/time support for NotesNote.replaceItemValue(...)
- New utility class to format view data as markdown table
- Added method to get agent design doc UNID
- Added hierarchical recycling (parent/child auto recycle)
Notes Workspace API
The new Notes workspace API is pretty feature complete. It let's you read and modify the workspace content and contains functionality that I have long been waiting for in the Notes Client user interface like a method to change the order of tabs.
Actually I built this for my own purpose because I needed a tool to remove icons that no longer exist.
To play with the API and as a sample for a standalone application using Domino JNA, I created a small Maven project that you can find here:
https://github.com/klehmann/domino-jna/tree/develop/workspace-demo
The application creates a new workspace page:
and a database icon with a red rectangle:
We support both the classic 32x32 pixel icons and the new 64x64 true color ones. By passing a NotesDatabase object to NotesWorkspace.addIcon(...), the current DB icon gets extracted from the NSF design and copied into the workspace.
Please make sure that the Notes Client is not running or in its early startup phase when you modify the workspace. Otherwise the changes will not be visible in the Notes UI and might later be overwritten.
Formula magic
Classic formula execution is limited to 64K of return data. We found a way to remove this limit (e.g. to use @DbColumn to read the whole column of a view) and you can even run formulas in a restricted environment, e.g. to prevent them from changing the Notes.ini or setting document items.
Methods have been added to analyze a formula in order to see if it's time variant, just contains a field name or uses special functions like @DocSiblings or @DocDescendants.
QueryResultsProcessor API
The QueryResultsProcessor lets you build views across NSF boundaries, have them displayed in the Notes Client or exported in JSON format as string.
In contrast to the implementation in the legacy Notes.jar and LotusScript API, we provide methods to get the JSON result back in a streaming way so that it can be parsed without storing the whole content in the Java heap.