We're delighted to release SequenceServer 3!

New features!

• Better reporting of BLAST database formatting errors. BLAST has become better at identifying and at reporting database errors early on... e.g., duplicate identifiers, illegal characters, or identifiers longer than 50 characters. These are now better reported by sequenceserver - #692 & #690
• Automatically detect and convert FASTQ to FASTA query inputs - #721 - You typically should avoid running BLAST on FASTQ files... but sometimes it is the right thing to do.
• Allow bigger sequence queries - #711 - We previously limited input queries to 10Mb. That could be frustrating when trying to BLAST/annotate a transcriptome or predicted geneset. By default we now support queries up to 250Mb. This is enabled by a recent change (2.1 or 2.2) which allows users to download table/xml results for huge results rather than having to load the full html report.
• SequenceServer Cloud users now get email notifications when long running jobs complete - #723

Maintenance, bugfixes and minor improvements

• Bump Sinatra dependency to v4 - #726 - we were previously on v2. This also means that our rack dependency shifted from v2 to v3. This is a major dependency change. Based on semantic versioning rules, these backwards-incompatible changes justify our bump from 2.20 to 3.0
• BLAST+ 2.15
• Set encoding defaults to UTF-8 - #694
• Results page tweaks - #699
• CSS paragraph clamping - #700
• Fix of #695 issue - #702
• Make response gzipping optional - #704
• Fix an issue when initializing SequenceServer with unformatted DBs - #708
• Fixed the job 404 exception error #666 - #710
• Drop webshim polyfill which was needed for old browsers - #718
• Add a dedicated 404 page - #719
• Re Sidebar Element Clickability Fix - #724
• And other minor changes... see the Full Changelog: v2.2.0...v3.0.1

We're ecstatic to release SequenceServer BLAST 2.2.0.

This further builds on some of the great new features and bug-fixes from 2.1.0. Notably:

• Quick as a Lightning Bolt! SequenceServer now launches much faster. This comes from two changes. First, we now only check for databases that need formatting when we intend to format c48cfc8. And you can now also supercharge your launch time even more when using the new --optimistic flag. This disables all checking and thus should only be done when the administrator is certain that all databases were formatted with appropriate makeblastdb parameters. Combined, our launch times on large databases have reduced from minutes to seconds. Thank you @tadast
• Download large text-format BLAST results without rendering. Sometimes users would run analyses that provided very large results (e.g., BLAST output of 250Mb or more). This can happen when, for example a user runs BLAST from a new geneset containing 10-15,000 genes against a large database, and requests hundreds of hits per query. Showing the result page would typically lead the browser (and sometimes SequenceServer) to crash. We now instead provide a warning, and allow the user to simply download the .xml or tab-delimited BLAST report. Thank you @tadast
  
• Format .faa. We now ensure that SequenceServer formats .faa files (this extension is occasionally used FASTA that contain amino-acid sequence). 1ee9171
• Error reporting. We now ensure that when database formatting fails, we keep track of errors. Credit @tadast 10ecaad
• Docker bump. Docker build is now from ruby:3.2.2-bullseye rather than slim-buster. This improves compatibility. 95ecc52 - thank you @nathanweeks
• Architectural rewiring - some tweaks improve code maintainability

Stay grounded with this supercharged release.

Full list of changes:

• Improve test coverage by @Lazy-poet in #657
• Use full image for builder stage; update to bullseye by @nathanweeks in #675
• Fix thread shutdown logic by @tadast in #681
• Handling large BLAST results by @tadast in #682
• Record makeblastdb errors for debugging purposes by @tadast in #683
• Add .faa as a recognised fasta extension by @tadast in #684
• [Performance] Do not perform unnecessary operations in makeblastdb by @tadast in #686
• [Performance] Add --optimistic flag that skips DB checks by @tadast in #687
• Gh actions grg by @augeas in #688

Improved bioinformatic analysis, collaboration, and user experience

1. Upgrade to BLAST 2.14. Among other things, this enables access to faster algorithms using arguments such as -task blastp-fast (20% to 300% faster depending on database size).

2. [New]: Enable cloud sharing. When running Sequenceserver on a local machine, it may be hard to share the results with anyone, unless they are looking at your screen over your shoulder :)
   Here, we enable sharing the results page with anyone who has access to the internet. Users have an option in the sidebar to share the results to Cloud, which will upload them to be hosted with a secret URL on http://sequenceserver.com/cloud for free.
   

3. [New]: Improved documentation of optional BLAST parameters #608.

4. Improved performance on slow connection (e.g., it was previously possible to click "BLAST" a second while waiting for server response). #526

Improved code quality, security, software compatibility and robustness

5. Migration of old javascript code to ES6. Similar updating to newer react (this took two steps. First updating to 0.14 which was the earliest to use ReactDOM, then updating to v18). This was a big chunk of work. These changes significantly improve maintainability, reliability and robustness. See #583 (and #556 for rationale)
6. Upgrade to Ruby 3.2 as 2.x are now deprecated and can include security risks.
7. We now also have js unit testing using the https://jestjs.io framework.
8. Many many improvements to robustness and to code maintainability.
9. Many bug fixes.
10. Improved security (see #569).
11. Migrated from TravisCI to Github actions for automated unit and integration testing.
12. Improvements in ruby/capybara testing robustness as rspec tests weren't necessarily waiting long enough for browser to render

1. Upgrade to BLAST+ 2.12.0
2. We rolled out a framework over the last two releases to deal with issues arising from the introduction of the new BLAST database format and databases created without the -parse_seqids option of makeblastdb. This took the form of SequenceServer trying to detect old format and non-parse_seqids databases on startup and forcing users to upgrade the databases to the new format or rebuild with -parse_seqids option. We are changing our approach. SequenceServer will still detect old format and non-parse_seqids database on startup, but it will no longer force users to upgrade or rebuild BLAST databases. Instead, it will display a warning for each such database and launch as before. For non-parse_seqids databases, SequenceServer will additionally disable FASTA download links on the results page.
   • To rebuild BLAST databases and enable FASTA download links, use sequenceserver -m.
   • We had previously noticed that mixing old and new format databases can cause BLAST searches to fail. We have not been able to replicate these results with BLAST+ 2.12.0. If you encounter any such issue, please try upgrading old format databases with sequenceserver -m and report the issue.
   • Upgrading or rebuilding databases can be slow for very large databases and fail if sequence ids are longer than 50 characters. The latter is a limitation of BLAST. sequenceserver -m informs users of these limitations and also advises to backup the databases before reformatting (inspired by Lukasz Sobala's experience).
   • Thanks to all the users who tried the initial approach and reported issues. Most of them are documented below. Special thanks to @Shellfishgene for being one of the first to critique the initial approach (#513).
3. The detection of old and new format databases has been revised to use the %v option of blastdbcmd -list_outfmt in BLAST+ 2.12.0. This is more robust than our initial approach which relied on file extensions and would miss some cases.
4. Database aliases created using blastdb_aliastool are now ignored by the database reformatting utility. This prevents a bug where the reformatting utility could 'corrupt' the larger database that was referenced by the alias (credit: Lukasz Sobala, #521; special thanks for providing a wonderfully detailed bug report).
5. Detection of non-parse_seqids option would fail in several cases (see issues #511, #512, #519). We have fixed the reported issues and an unreported issue with multi-part databases.
6. Attempting to reformat databases created without the -parse_seqids option using sequenceserver -m, would end in the following error message “Error: [makeblastdb] No sequences matched any of the taxids provided”. Fixed it.
7. SequenceServer should continue its startup routine after offering to create BLAST databases during the initial setup, but a programming error prevented that and it would instead quit after creating BLAST databases. Fixed it.
8. Fixed a rare issue where SequenceServer would classify a non-FASTA file as FASTA during initial setup (or when using sequenceserver -m) and offer to create BLAST databases from it (#567).
9. Fixed a rare issue where SequenceServer would fail to scan databases directory during startup (or when using sequenceserver -m) when running inside a docker container.
10. SequenceServer asks for an optional taxonomy id during database creation, however, a programming error accidentally made it compulsory (sorry!). We have made it optional again.
11. The plugin/extensions file (-r option) is now loaded before searching for BLAST binaries and databases so that these aspects of SequenceServer can also be customised through the extensions file for advanced use cases (credit: Elvin).
12. Ability to drag and drop a FASTA file to the search form was not working in Safari. Fixed it.
13. The form state was not correctly restored in some cases when using the browser's back button or the ‘Edit search’ link on the results page. Fixed it.
14. The database 'Select all' button would get stuck if a database was manually selected first and the 'Select all' was subsequently used (#562). Fixed it. Thanks to @MatthAlex for reporting the issue.
15. We have introduced an optional, experimental widget to display databases in tree format - very helpful if you have a long list of databases. The feature was contributed by @Bjoernsen (#520) and is similar to what was previously produced by lepbase (#307). See #520 for details. To activate the widget, add ":databases_widget: tree" to your SequenceServer config file (which is ~/.sequenceserver.conf by default). The tree widget additionally allows linking databases to external page (how to do so will be documented on the website in due time)
16. Commas are now allowed in advanced params input so that multiple values can be provided to options such as -taxids (credit: Lukasz Sobala)
17. Include an entry for the very helpful -task option to the handy reference of command-line BLAST options included in the search form - accessed using the ? button next to advanced params input field (#517).
18. Removed the twitter button from the footer and made the footer more compact.
19. In the results page we have reverted to showing identity of top HSP in the per-query hit table instead of the average identity of all HSPs for the matching database sequence (credit: Etienne Bucher, #506).
20. HSPs in Kablammo visualisation are no longer labelled if there are too many HSPs, as too many labels creates clutter without adding much value (#518).
21. SequenceServer used to automatically hide sidebar containing download links etc. if the BLAST search resulted in no hits. However, as the sidebar also contains the links to edit or start a new search since the last few candidate releases, the sidebar is now always shown.
22. Ensure long query sequence ids wrap in the sidebar instead of shooting past the sidebar boundary (#571). Credit @MatthAlex
23. The links to download FASTA and pairwise alignment of all hits are now disabled if the BLAST search results in no hits (#552).
24. We have introduced a new num_jobs setting. This is the number of concurrent BLAST searches that SequenceServer will run - the default value is 1. This is distinct from num_threads, which is passed to BLAST, and is the number of threads that each BLAST job will use.
25. Made sure that SequenceServer will write default values of new configuration options to the configuration file for new users. Existing users can update their configuration file to see all possible configuration values by running sequenceserver -s
26. The whichdb function, that can be used in link generators to determine which database a hit came from, was returning CommandFailed error in some cases. Fixed it (credit: @jveera888, #529).
27. If you like to customise SequenceServer and use our docker image, there is now an option to build JS and CSS assets as part of the docker build step: just add --target=minify to your docker build command. Note that this requires BuildKit to be enabled (credit: Nathan Weeks)
28. Huge shout-out to @Iain-S for fixing several code style issues (JS and CSS), and revising our code linting framework (#531, #532).

Features

• Automatically check for incompatible databases on startup and prompt users to reformat them.
• Reformatting databases now preserves taxonomy information embedded in the database (if any).
• Add ability to use -taxids_map of makeblastdb during database creation. To use it place a '.taxids_map.txt' file next to the FASTA file.
• Add 'Edit search' and 'New search' links to the report page (thanks to Tomas-Pluskal & TomMD for the push).
• Add option to open BLAST results in a new tab.
• Search form can now be cleared by reloading the page. Relevant if you used browser's back button or 'Edit search' link and wanted to clear the form to start over.
• Make it easier to pass command line arguments to Docker image. For example, number of BLAST threads can now be set as docker run ... wurmlab/sequenceserver sequenceserver -n 4 instead of docker run ... wurmlab/sequenceserver bundle exec bin/sequenceserver -d /db -n4 (yeah!)
• Reduce size of Docker image 3-fold (credit: Nathan Weeks, Neel Kamath)
• Reduce size of gem file 3-fold (credit: Nathan Weeks)

Bug fixes

• Fix a bug that prevented query textarea from being resized veritcally (credit: Iain-S).
• Fix a bug that prevented the use of keyboard shortcut 'Ctrl-Enter' to submit the search.
• Fix a bug that would prevent search form from rendering query sequences sent to it via POST request (credit: Sam Hokin).
• Fix a bug that would cause an infinite loop on search form if the user had only one database.
• Fix a bug that could cause new users to be prompted more than once to join the announcements list.

• Switch to using sequence id instead of accession for sequence retrieval. This fixes FASTA download for gnl|Morex|chr type sequence ids when using version 5 database (#475). Thanks to Eric Y for reporting the issue.
• Ensure SequenceServer does not crash if it could not determine host IP, which can be the case if it is being run offline (#482). Thanks to Vladimir for reporting the issue.
• Ensure error modal does not remove the rest of the content from the page.

• JSON endpoints responded with content-type 'plain/text'. Change that to 'application/json'. Thanks to Richard Adams for reporting the issue.
• Include hit title (the stuff after sequence id) in the summary table of hits per query. Of course, titles can be really long, so the text is truncated with ... when it overflows the table cell. The entire title is displayed in a tooltip. Feature requested by Niek Art.

1. Running sequenceserver -m will now automatically detect older V4 databases, and those created without the -parse_seqids option of makeblastdb, and offer to rebuild them. This works even if you deleted the original FASTA files. Database titles are preserved when rebuilding, however, taxonomy information in the database is unfortunately not preserved.
2. BLASTing a mix of older V4 and newer V5 databases causes error. SequenceServer now catches and informs the user of this error. Thanks to Massimiliano babbucci for reporting this issue.
3. The list of databases in the search form should be alphabetically sorted. This behaviour was lost in the rewrite leading to version 2.0 and has now been fixed. Thanks to Loraine Guéguen for reporting the issue.

• Make it possible to run with BLAST+ 2.9.0 (#463)
• Print diagnostic information when makeblastdb fails (#469)
• Do not prompt to join mailing list when running under Docker (#468)
• Update docker base image and reduce docker image size by ~100 Mb

• Improve regular expression used to detect multi-part databases, so that databases that aren't multi-part are recognised and displayed in the interface (#465). Issue reported and fixed by Loraine Brillet-Guéguen.