Comparison of re-Isearch to Popular Engines
Abstract
There are a vast number of engines on the market. On the commerical side there is, for example, MarkLogic; there are RBMSs with fulltext support like SQLlite and the extraordinary PostgreSQL; and on the “open source” side there engines like ElasticSearch, Solr and Neo4j. The last 3, interestingly, are not really engines but are applications based around the Lucene engine. This study thus compares re-Isearch to Lucene.
Apache Solr
Apache Solr is an open-source search server built on top of Apache Lucene that provides all of Lucene’s search capabilities through HTTP requests. It has been around for almost a decade and a half, making it a mature product with a broad user community.
Solr uses request handlers to ingest data from XML files, CSV files, databases, Microsoft Word documents, and PDFs. It provides native support for the Apache Tika library.
A Solr configuration requires at least 512 MB of HEAP memory to allocate to instances.
Solr is licensed with the Apache 2.0 License. It is an official Apache project.
ElasticSearch
Elasticsearch is also an open-source search engine built on top of Apache Lucene. It extends Lucene’s indexing and search functionalities using RESTful APIs, and it archives the distribution of data on multiple servers using the index and shards concept. Elasticsearch is completely based on JSON. It supports data ingestion from multiple sources using the Beats family (lightweight data shippers available in the ELK Stack) and Logstash.
The default Elasticsearch configuration requires 1GB of HEAP memory.
Elastic started off under the Apache 2.0 license but is now dual licensed under both the Elastic License and SSPL. While the former disallows providing the software to third parties as a hosted or managed service, SSPL allows it but requires anyone who wants to offer ElasticSearch as a service to either release all surrounding infrastructure as SSPL or get a commercial license.
MarkLogic
MarkLogic is a commerical search system originally developed for XML. It indexes the content and structure of documents including words, phrases, relationships, and values in over 200 languages with tokenization, collation, and stemming for core languages. It supports search across its data and metadata using a word or phrase and incorporates Boolean logic, stemming, wildcards, case sensitivity, punctuation sensitivity, diacritic sensitivity, and search term weighting.
Rust Based
1) Tantivy: https://github.com/quickwit-oss/tantivy
Tantivy is a full text search engine library written in Rust.
It is closer to Apache Lucene than to Elasticsearch or Apache Solr in the sense it is not an off-the-shelf search engine server, but rather a crate that can be used to build such a search engine.
Tantivy is, in fact, strongly inspired by Lucene's design.
2) Bayard: https://github.com/bayard-search/bayard
Bayard is a full-text search and indexing server written in Rust built on top of Tantivy that implements Raft Consensus Algorithm and gRPC.
3) Toshi: https://github.com/toshi-search/Toshi
Toshi is meant to be a full-text search engine similar to Elasticsearch. Toshi strives to be to Elasticsearch what Tantivy is to Lucene.
4) Sonic: https://github.com/valeriansaliou/sonic
Sonic is a fast, lightweight and schema-less search backend. It ingests search texts and identifier tuples that can then be queried against in a microsecond's time.
Sonic can be used as a simple alternative to super-heavy and full-featured search backends such as Elasticsearch in some use-cases. It is capable of normalizing natural language search queries, auto-completing a search query and providing the most relevant results for a query. Sonic is an identifier index, rather than a document index; when queried, it returns IDs that can then be used to refer to the matched documents in an external database.
5) Vector https://vector.dev/ and https://github.com/vectordotdev/vector
Alternative to Elastic’s Logstash
Angolia
https://en.wikipedia.org/wiki/Algolia
Algolia was founded in 2012 by Nicolas Dessaigne and Julien Lemoine, both originally from Paris, France. It was originally a company focused on offline search on mobile phones. Later it was selected to be part of Y Combinator's Winter 2014 class.
The Algolia model provides search as a service, offering web search across a client's website using an externally hosted search engine
“Subscribe for free, pay based on your usage”
Typesense
https://github.com/typesense/typesense
Core Overview
| Typesense | Algolia | ElasticSearch | Meilisearch |
Source Code | Fully open source | Proprietary closed source | Source-available, licensed under SSPL | Fully open source |
First Commit | 2015 | 2012 | 2010 | 2018 |
Built Using | C++ | C++ | Java | Rust |
Core Search Algorithm | Built from the ground-up | Built from the ground-up | Built on top of Lucene | Built from the ground-up |
Best Suited For | Instant Search-as-you-type Experiences for data sets that can fit in RAM, up to 24 TB (or current commercially available RAM size). | Instant Search-as-you-type Experiences for datasets up to 128 GB in size. | General-purpose search & aggregations over petabyte-scale datasets (eg: log data) | Instant Search-as-you-type Experiences for up to a few hundred thousand records, that don't require a production-grade highly-available setup. |
Primary Index Location | RAM | RAM | Disk, with RAM cache | Disk with Memory Mapped files |
Re-Isearch
Re-Isearch is a reborn open-source search engine built on the basis of the Isearch open-source engine and IB (a proprietary fork). It indexes the content and structure of documents including words, phrases, relationships, and values. It supports search across its data and metadata using a word or phrase and incorporates Boolean logic, wildcards, case sensitivity, punctuation sensitivity, diacritic sensitivity, and search term weighting.
The default configuration requires a minimum heap size 3x the size of the largest document it intends to index. It has run well on machines with virtual memory and as little as 8 MB RAM.
The engine has, in various forms (and using differing algorithms), been available since the 1990s and was originally developed to provide structured field search via the NISO Z39.50 protocol (an international standard client–server, application layer communications protocol for searching and retrieving information from a database that dominates, for example, the library world) . Its well established user community was primarily in the public sector and included many high profile projects such as the U.S. Patent and Trademark Office (USPTO) patent search, the Federal Geographic Data Clearinghouse (FGDC), the NASA Global Change Master Directory, the NASA EOS Guide System, the NASA Catalog Interoperability Project, the astronomical pre-print service based at the Space Telescope Science Institute, The PCT Electronic Gazette at the World Intellectual Property Organization (WIPO), the SAGE Project of the Special Collections Department at Emory University, Eco Companion Australasia (an environmental geospatial resources catalog), Australian National Genomic Information Service (ANGIS), the Open Directory Project and numerous governmental portals.
The engine has a number of extensions that enables search functionality via ISO23950/Z39.50, OASIS SRU/W and OpenSearch REST API.
Re-Isearch is licensed under the Apache 2.0 license. The Z39.50, SRU/W and other servers are licensed under highly permissive MIT style licenses. Basically these means anyone has permission to use, copy, modify, distribute, and sell the software and its documentation, in whole or in part, for any purpose without fee.
re-Isearch | MarkLogic | Elasticsearch | Apache Solr |
NoSQL search engine | Operational and transactional Enterprise NoSQL database | A distributed, RESTful modern search and analytics engine based on Apache Lucene | A widely used distributed, scalable search engine based on Apache Lucene |
NativeXML DBMS, RDF Store, search engine | NativeXML DBMS, RDF Store, search engine | Search engine | Search engine |
Object DBMS including Spatial |
| Document store Spatial DBMS | Spatial DBMS |
1994-2011, reborn 2021 | Since 2001 | Since 2010 | Since 2006 |
C++ | C++ | Java | Java |
Free | Commerical | Partially Free | Free |
Open Source | Proprietary | Open Source | Open Source |
XML support | XML support | JSON Only | XML support |
Foreign keys, Join | No foreign keys | No foreign keys | No foreign keys |
Schema-Free | Schema-Free | Schema-Free | Schema |
Multi-language API, Z39.50, SRU/W. CQL, IB Query Language,... | Multi-language API, Xquery, SPARQL, | Java API RESTful HTTP/JSON API | Java API RESTful HTTP/JSON API |
Search during index | Search during index | No | No |
Own algorithms | Own algorithms | Based on Lucene (inverted index) | Based on Lucene (inverted index) |
Relation DB based
MantiCore Search
https://github.com/manticoresoftware/manticoresearch
Manticore Search is a multi-storage database designed specifically for search, including full-text search.
Craigslist, Socialgist, PubChem and many others use Manticore for efficient searching and stream filtering.
Manticore Search was forked from Sphinx 2.3.2 in 2017.
Sphinx
Sphinx can be used either as a stand-alone server or as a storage engine ("SphinxSE") for the MySQL family of databases. When run as a standalone server Sphinx operates similar to a DBMS and can communicate with MySQL, MariaDB and PostgreSQL through their native protocols or with any ODBC-compliant DBMS via ODBC. MariaDB, a fork of MySQL, is distributed with SphinxSE.
Lucene
pageLucene vs. re-IsearchLast updated