Everything you need to get started with Badger
$GOBIN
path.
DB
. It represents multiple files on disk
in specific directories, which contain the data for a single database.
To open your database, use the badger.Open()
function, with the appropriate
options. The Dir
and ValueDir
options are mandatory and you must specify
them in your client. To simplify, you can set both options to the same value.
InMemory
option.
100 Mb
cache:
DB.View()
method:
DB.Update()
method:
ErrConflict
error is reported in case of a conflict. Depending on the state
of your app, you have the option to retry the operation if you receive this
error.
An ErrTxnTooBig
is reported in case the number of pending writes/deletes in
the transaction exceeds a certain limit. In that case, it’s best to commit the
transaction and start a new transaction immediately. Here is an example (we
aren’t checking for errors in some places for simplicity):
DB.View()
and DB.Update()
methods are wrappers around the
DB.NewTransaction()
and Txn.Commit()
methods (or Txn.Discard()
in case of
read-only transactions). These helper methods start the transaction, execute a
function, and then safely discard your transaction if an error is returned. This
is the recommended way to use Badger transactions.
However, sometimes you may want to manually create and commit your transactions.
You can use the DB.NewTransaction()
function directly, which takes in a
boolean argument to specify whether a read-write transaction is required. For
read-write transactions, it’s necessary to call Txn.Commit()
to ensure the
transaction is committed. For read-only transactions, calling Txn.Discard()
is
sufficient. Txn.Commit()
also calls Txn.Discard()
internally to cleanup the
transaction, so just calling Txn.Commit()
is sufficient for read-write
transaction. However, if your code doesn’t call Txn.Commit()
for some reason
(for e.g it returns prematurely with an error), then please make sure you call
Txn.Discard()
in a defer
block. Refer to the code below.
DB.NewTransaction()
is a boolean stating if the
transaction should be writable.
Badger allows an optional callback to the Txn.Commit()
method. Normally, the
callback can be set to nil
, and the method returns after all the writes have
succeeded. However, if this callback is provided, the Txn.Commit()
method
returns as soon as it has checked for any conflicts. The actual writing to the
disk happens asynchronously, and the callback is invoked once the writing has
finished, or an error has occurred. This can improve the throughput of the app
in some cases. But it also means that a transaction isn’t durable until the
callback has been invoked with a nil
error value.
Txn.Set()
method:
Entry
, then setting this
Entry
using Txn.SetEntry()
. Entry
also exposes methods to set properties
on it.
"answer"
key to "42"
. To retrieve this value, we
can use the Txn.Get()
method:
Txn.Get()
returns ErrKeyNotFound
if the value isn’t found.
Please note that values returned from Get()
are only valid while the
transaction is open. If you need to use a value outside of the transaction then
you must use copy()
to copy it to another byte slice.
Use the Txn.Delete()
method to delete a key.
DB.GetSequence
method. This method returns a Sequence
object, which
is thread-safe and can be used concurrently via various goroutines.
Badger would lease a range of integers to hand out from memory, with the
bandwidth provided to DB.GetSequence
. The frequency at which disk writes are
done is determined by this lease bandwidth and the frequency of Next
invocations. Setting a bandwidth too low would do more disk writes, setting it
too high would result in wasted integers if Badger is closed or crashes. To
avoid wasted integers, call Release
before closing Badger.
MergeFunc
which takes in an existing value, and a value to be merged
with it. It returns a new value which is the result of the merge operation. All
values are specified in byte arrays. For example, this is a merge function
(add
) which appends a []byte
value to an existing []byte
value.
DB.GetMergeOperator()
method, along
with a key, and a duration value. The duration specifies how often the merge
function is run on values that have been added using the MergeOperator.Add()
method.
MergeOperator.Get()
method can be used to retrieve the cumulative value of the
key associated with the merge operation.
time.Duration
value using the
Entry.WithTTL()
and Txn.SetEntry()
API methods.
Entry.WithMeta()
and Txn.SetEntry()
API methods.
Entry
APIs can be used to add the user metadata and TTL for same key. This
Entry
then can be set using Txn.SetEntry()
.
Iterator
, which can be obtained using the
Txn.NewIterator()
method. Iteration happens in byte-wise lexicographical
sorting order.
IteratorOptions.PrefetchSize
field. However, setting it to a
value higher than GOMAXPROCS
(which we recommend to be 128 or higher)
shouldn’t give any additional benefits. You can also turn off the fetching of
values altogether. See section below on key-only iteration.
Seek()
and ValidForPrefix()
:
feed:userUuid:timestamp:postUuid
. Here, the
timestamp
part of the key is treated as an attribute, and items are stored in
the corresponding order:
Order Ascending | Key |
---|---|
1 | feed:tQpnEDVRoCxTFQDvyQEzdo:1733127889:tQpnEDVRoCxTFQDvyQEzdo |
2 | feed:tQpnEDVRoCxTFQDvyQEzdo:1733127533:1Mryrou1xoekEaxzrFiHwL |
3 | feed:tQpnEDVRoCxTFQDvyQEzdo:1733127486:pprRrNL2WP4yfVXsSNBSx6 |
feed:tQpnEDVRoCxTFQDvyQEzdo
. All matching keys are returned, sorted by
timestamp
.timestamp
or
reversed timestamp
as needed:
IteratorOptions.PrefetchValues
field to false
. This can also be used to do
sparse reads for selected keys during an iteration, by calling item.Value()
only when required.
ChooseKey(item)
, followed by KeyToList(key, itr)
. This allows a user to
select or reject that key, and if selected, convert the value versions into
custom key-values. The goroutine batches up 4 MB worth of key-values, before
sending it over to a channel. Another goroutine further batches up data from
this channel using smart batching algorithm and calls Send
serially.
This framework is designed for high throughput key-value iteration, spreading
the work of iteration across many goroutines. DB.Backup
uses this framework to
provide full and incremental backups quickly. Dgraph is a heavy user of this
framework. In fact, this framework was developed and used within Dgraph, before
getting ported over to Badger.
DB.RunValueLogGC()
: This method is designed to do garbage collection while
Badger is online. Along with randomly picking a file, it uses statistics
generated by the LSM tree compactions to pick files that are likely to lead to
maximum space reclamation. It is recommended to be called during periods of
low activity in your system, or periodically. One call would only result in
removal of at max one log file. As an optimization, you could also immediately
re-run it whenever it returns nil error (indicating a successful value log
GC), as shown below.
DB.PurgeOlderVersions()
: This method is DEPRECATED since v1.5.0. Now,
Badger’s LSM tree automatically discards older/invalid versions of keys.
RunValueLogGC
method would not garbage collect the latest value log.DB.Backup()
and DB.Load()
which can be used
to do online backups and restores. Badger v0.9 provides a CLI tool badger
,
which can do offline backup/restore. Make sure you have $GOPATH/bin
in your
PATH to use this tool.
The command below creates a version-agnostic backup of the database, to a file
badger.bak
in the current working directory
badger.bak
in the current working directory to a new database:
badger --help
for more details.
If you have a Badger database that was created using v0.8 (or below), you can
use the badger_backup
tool provided in v0.8.1, and then restore it using the
preceding command to upgrade your database to work with the latest version.
Backup
and Restore
APIs and tools.
However, Badger is also rsync-friendly because all files are immutable, barring
the latest value log which is append-only. So, rsync can be used as rudimentary
way to perform a backup. In the following script, we repeat rsync to ensure that
the LSM tree remains consistent with the MANIFEST file while doing a full
backup.
Options
struct that’s passed in when opening the database using DB.Open
.
Options.NumMemtables
)
Options.NumMemtables
, also adjust
Options.NumLevelZeroTables
and Options.NumLevelZeroTablesStall
accordingly.Options.NumCompactors
)Options.BaseTableSize
)Options.ValueLogFileSize
)