Greetings GenBank Users,
GenBank Release 180.0 is now available via FTP from the National
Center for Biotechnology Information (NCBI):
Ftp Site Directory Contents
---------------- --------- ---------------------------------------
ftp.ncbi.nih.gov genbank GenBank Release 180.0 flatfiles
ncbi-asn1 ASN.1 data used to create Release 180.0
Close-of-data for GenBank 180.0 occurred on 10/15/2010. Uncompressed,
the Release 180.0 flatfiles require roughly 457 GB (sequence files only)
or 490 GB (including the 'short directory', 'index' and the *.txt
files). The ASN.1 data require approximately 375 GB.
Recent statistics for non-WGS, non-CON sequences:
Release Date Base Pairs Entries
179 Aug 2010 117476523128 122941883
180 Oct 2010 118551641086 125764384
Recent statistics for WGS sequences:
Release Date Base Pairs Entries
179 Aug 2010 169253846128 58994334
180 Oct 2010 175339059129 59397637
During the 60 days between the close dates for GenBank Releases 179.0
and 180.0, the non-WGS/non-CON portion of GenBank grew by 1,075,117,958
basepairs and by 2,822,501 sequence records. During that same period,
464,050 records were updated. An average of about 54,775 non-WGS/non-CON
records were added and/or updated per day.
Between releases 179.0 and 180.0, the WGS component of GenBank grew by
6,085,213,001 basepairs and by 403,303 sequence records.
For additional release information, see the README files in either of
the directories mentioned above, and the release notes (gbrel.txt) in
the genbank directory. Sections 1.3 and 1.4 of the release notes
(Changes in Release 180.0 and Upcoming Changes) have been appended
below for your convenience.
** Important Notes **
* GenBank 'index' files are now provided without any EST content, and
without most GSS content. See Section 1.3.3 of the release notes for
further details.
NCBI is considering ceasing support for the index files, so we
encourage affected users to review that section and provide feedback.
Release 180.0 data, and subsequent updates, are available now via
NCBI's Entrez and Blast services.
As a general guideline, we suggest first transferring the GenBank
release notes (gbrel.txt) whenever a release is being obtained. Check
to make sure that the date and release number in the header of the
release notes are current (eg: October 15 2010, 180.0). If they are
not, interrupt the remaining transfers and then request assistance from
the NCBI Service Desk.
A comprehensive check of the headers of all release files after your
transfers are complete is also suggested. Here's how one might go about
this on a Unix or Linux platform, using csh/tcsh :
set files = `ls gb*.*`
foreach i ($files)
head -10 $i | grep Release
end
Or, if the files are compressed, perhaps:
gzcat $i | head -10 | grep Release
If you encounter problems while ftp'ing or uncompressing Release
180.0, please send email outlining your difficulties to:
info from ncbi.nlm.nih.gov
Mark Cavanaugh, Michael Kimelman, Ilya Dondoshansky, Sergey Zhdanov
GenBank
NCBI/NLM/NIH/HHS
1.3 Important Changes in Release 180.0
1.3.1 Organizational changes
The total number of sequence data files increased by 26 with this release:
- the BCT division is now composed of 60 files (+1)
- the CON division is now composed of 142 files (+2)
- the ENV division is now composed of 30 files (+1)
- the EST division is now composed of 422 files (+3)
- the GSS division is now composed of 217 files (+2)
- the HTG division is now composed of 136 files (+1)
- the INV division is now composed of 25 files (+1)
- the PAT division is now composed of 128 files (+7)
- the PLN division is now composed of 43 files (+1)
- the PRI division is now composed of 42 files (+1)
- the SYN division is now composed of 3 files (+1)
- the TSA division is now composed of 12 files (+4)
- the VRT division is now composed of 22 files (+1)
The total number of 'index' files increased by 6 with this release:
- the AUT (author name) index is now composed of 80 files (+5)
- the JOU (journal name) index is now composed of 10 files (+1)
1.3.2 /artificial_location is now a controlled-value qualifier
The /artificial_location is now a controlled-vocabulary qualifier,
with values of "heterogenous population sequenced" or "low-quality
sequence region", as of the October 2010 GenBank Release.
1.3.3 Discontinued qualifiers: /codon and /label
With October's GenBank Release 180.0, both the /codon and /label
qualifiers have been discontinued.
Existing /codon qualifiers have been replaced by the appropriate
/transl_except qualifier. If the /codon qualifier impacted multiple
protein residues of a coding region's translation, a /transl_except
has been added for each such residue.
The text from existing /label qualifiers have been moved to /note
qualifers of the features involved.
1.3.4 Discontinued feature: conflict
The conflict feature has been discontinued as of the October 2010
GenBank Release. Existing conflict features have been converted to
misc_difference (which supports all of the qualifiers that conflict
used to).
1.3.5 New feature: mobile_element
A new feature for the annotation of gene cassettes, transposons, and
insertion sequences is now supported as of this October 2010 GenBank
Release: mobile_element .
This new feature has a mandatory /mobile_element_type qualifier, to
provide information about the nature of the mobile element. At present,
this is a free-text qualifier.
Existing repeat_region features that possessed /mobile_element qualifiers
have been converted to mobile_element features. All qualifiers other than
/mobile_element which have been legal for repeat_region are also supported
for mobile_element, and they were moved to the new feature during the
conversion.
We expect that the /mobile_element qualifier will be discontinued within
a fairly short time.
1.3.6 Three previously-announced changes delayed
Three of the Feature-Table changes that were agreed to at the May 2010
International Nucleotide Sequence Database Collaboration annual meeting,
and scheduled for October 2010 implementation, will be delayed. These
changes are:
Discontinuation of the /partial qualifier.
A classification system for /inference and /experiment.
Introduction of the /whole_replicon qualifier.
See Section 1.4 (Upcoming Changes) for further details.
1.3.7 Changes in the content of index files
As described in the GB 153 release notes, the 'index' files which accompany
GenBank releases (see Section 3.3) are considered to be a legacy data product by
NCBI, generated mostly for historical reasons. FTP statistics from January 2005
seem to support this: the index files were transferred only half as frequently as
the files of sequence records. The inherent inefficiencies of the index file
format also lead us to suspect that they have little serious use by the user
community, particularly for EST and GSS records.
The software that generated the index file products received little
attention over the years, and finally reached its limitations in
February 2006 (Release 152.0). The required multi-server queries which
obtained and sorted many millions of rows of terms from several different
databases simply outgrew the capacity of the hardware used for GenBank
Release generation.
Our short-term solution is to cease generating some index-file content
for all EST sequence records, and for GSS sequence records that originate
via direct submission to NCBI.
The three gbacc*.idx index files continue to reflect the entirety of the
release, including all EST and GSS records, however the file contents are
unsorted.
These 'solutions' are really just stop-gaps, and we will likely pursue
one of two options:
a) Cease support of the 'index' file products altogether.
b) Provide new products that present some of the most useful data from
the legacy 'index' files, and cease support for other types of index data.
If you are a user of the 'index' files associated with GenBank releases, we
encourage you to make your wishes known, either via the GenBank newsgroup,
or via email to NCBI's Service Desk:
info from ncbi.nlm.nih.gov
Our apologies for any inconvenience that these changes may cause.
1.3.8 GSS File Header Problem
GSS sequences at GenBank are maintained in two different systems, depending
on their origin, and the dumps from those systems occur in parallel. Because
the second dump (for example) has no prior knowledge of exactly how many GSS
files will be dumped by the first, it does not know how to number its own
output files.
There is thus a discrepancy between the filenames and file headers for
ninety-four of the GSS flatfiles in Release 180.0. Consider gbgss124.seq :
GBGSS1.SEQ Genetic Sequence Data Bank
October 15 2010
NCBI-GenBank Flat File Release 180.0
GSS Sequences (Part 1)
87146 loci, 64102517 bases, from 87146 reported sequences
Here, the filename and part number in the header is "1", though the file
has been renamed as "124" based on the number of files dumped from the other
system. We hope to resolve this discrepancy at some point, but the priority
is certainly much lower than many other tasks.
1.4 Upcoming Changes
A variety of changes affecting the GenBank flatfile format were implemented
in October 2010, as a result of discussions among DDBJ, EBI, and NCBI at the
May 2010 International Nucleotide Sequence Database Collaboration annual
meeting.
However, three of the planned changes could not be completed for this
October 15 GenBank 180.0 release:
a) The /partial qualifier will be discontinued, given that 5' and 3'
partiality can be indicated via a feature's location ('<' and '>').
b) A classification system will be introduced for the /inference and
/experiment qualifiers, such that the inferential or experimental
information can be associated with a feature's existence, location,
or description.
c) A /whole_replicon qualifier will be introduced, allowing records
which represent the entirety of a replicon (a chromosome; an organelle;
a mitochondrial plasmid) to be identified as such.
An investigation of (a) for older sequence records has thrown into question
whether the /partial qualifier can actually be done away with. On-going
review over the next few weeks will determine whether this change will be
implemented as agreed, amended, or abandoned.
The implementation of (b) has been delayed because of delays in software
development supporting the new classification system. However, those delays
are short-lived and we expect that the change will be completed well before
the December 2010 GenBank release.
Specification of the conditions under which the /whole_replicon qualifier
is, and is not, allowed to be used for sequence records is not yet complete,
so implementation of (c) will likewise be delayed for a number of weeks.
We will inform GenBank users when the issues impacting these three changes
have been clarified.
