Database Reference
In-Depth Information
Yes, I always recommend that you use a script to execute pgloader. It makes it much easier to
iterate towards something that works. Loads never work first time, except in the movies.
Let's look at a typical example:
cookbook_pgloader.conf
[pgsql]
host = 192.168.0.5
base = pgloader
user = dim
log_file = /tmp/pgloader.log
log_min_messages = DEBUG
client_min_messages = WARNING
lc_messages = C
client_encoding = 'utf-8'
copy_every = 10000
null = ""
empty_string = "\ "
max_parallel_sections = 4
[load]
table = load_example
filename = simple/simple.data
format = csv
datestyle = dmy
field_sep = |
trailing_sep = True
columns = a:1, b:3, c:2
reject_log = /tmp/simple.rej.log
reject_data = /tmp/simple.rej
section_threads = 4
We can use the load script like the following:
pgloader --summary --vacuum --config cookbook_pgloader.conf
How it works...
pgloader copes gracefully with errors.
copy
loads all rows in a single transaction, so only a
single error is enough to abort the load. pgloader breaks down an input file into reasonably
sized chunks, and loads them piece-by-piece. If some rows cause errors, then pgloader will
iteratively check them so that it can skip those bad rows.
