Date   

Re: Extremely slow performance with shapefile and geopackage

René Buffat
 

Hi David

First I would check if you have a sufficient amount of RAM available. If not, this could explain the slow performance.
If this is the case, I would recommend to read, process and write the data in batches.

Otherwise, there are a lot of parameters that can impact the performance. E.g. how complex the geometries are, how many rows you want to write, how many parallel reads and write you have to the disk, etc.

Regarding geometries problems, I'm not entirely sure what you mean. But regardless, with big datasets, it's always a good option to debug with smaller datasets (e.g. the first thousand lines) and then test if everything works. 

And fully unrelated, I would recommend to use os.path.join(datadir, "data.shp") instead of data_dir+"data.gpkg"

lg rene


Extremely slow performance with shapefile and geopackage

davideps@...
 

Hi folks,

I loaded a 5GB CSV file, turned lat/long into points, and have tried to save as shapefile and geopackage. At first, I ran into problems with datetime and tuple columns, which I've since turned to STR or dropped entirely. However, after an hour I still was not able to save. Lines like:
geodata.to_file(filename=data_dir+"data.shp"driver = 'ESRI Shapefile')
geodata.to_file(data_dir+"data.gpkg"driver="GPKG")
seem to hang with empty files written to disk. When I kill the line in the interpreter, sometimes data gets written at the last minute--sometimes not. Are there common dateframe or geometry problems I should check first? 

-david


 


Fiona 1.8.10

Sean Gillies
 

Hi all, 

1.8.10 is on PyPI now. A new Windows bug has been found and so we'll have a 1.8.11 release early tomorrow. 


Re: Fiona 1.8.9 and GDAL 3

René Buffat
 

Dropping old versions of GDAL and Python is planned for Fiona 2.x. You can find more details under the following link: https://github.com/Toblerity/fiona-rfc/blob/master/rfc/0001-fiona-2-0-changes.md


Re: Fiona 1.8.9 and GDAL 3

Howard Butler
 



On Fri, Oct 18, 2019 at 4:06 PM Sean Gillies <sean.gillies@...> wrote:
Hi all,

Lack of support from the fiona package for GDAL 3.0 has become unbearable. We won't make you wait for 1.9.0 and will instead have a 1.8.9 release that is compatible with GDAL versions 1.11 through 3.0. The planned release date is next Monday, 2019-10-21.

Please note that the wheels published to PyPI for 1.8.9 will contain GDAL version 2.4.2. If you require GDAL 3.0, you will need to install it separately and install the non-binary fiona package (like `python -m pip install --no-binary fiona fiona`) or use conda and install fiona from the conda-forge channel, which is more than ready to drop support for GDAL 2.x.

Switching the PyPI wheels over to GDAL 3.0 and PROJ 6.x is a bit of a lift and is not yet scheduled. It may be part of the 1.9.0 release.

Thanks for your patience!

Sean,

The new GDAL 3.0 PR works for me and should provide a good stopgap. Have you thought about how far back into GDAL that Fiona will support? Ditching all the 1.x stuff would throw off some burden. The GDAL Barn stuff is likely to leak into Fiona if it tries to support both old and new as well. I'm sure you've run into these issues with rasterio already, however, so maybe Fiona will follow suit with the strategy you're using to handle the transition there.

Howard


Fiona 1.8.9 and GDAL 3

Sean Gillies
 

Hi all,

Lack of support from the fiona package for GDAL 3.0 has become unbearable. We won't make you wait for 1.9.0 and will instead have a 1.8.9 release that is compatible with GDAL versions 1.11 through 3.0. The planned release date is next Monday, 2019-10-21.

Please note that the wheels published to PyPI for 1.8.9 will contain GDAL version 2.4.2. If you require GDAL 3.0, you will need to install it separately and install the non-binary fiona package (like `python -m pip install --no-binary fiona fiona`) or use conda and install fiona from the conda-forge channel, which is more than ready to drop support for GDAL 2.x.

Switching the PyPI wheels over to GDAL 3.0 and PROJ 6.x is a bit of a lift and is not yet scheduled. It may be part of the 1.9.0 release.

Thanks for your patience!

--
Sean Gillies


1.8.8-1 update for fiona wheels on PyPI

Sean Gillies
 

Hi all,

The original wheels for 1.8.8 that I uploaded on 2019-09-25 lacked libsqlite3 shared libraries and were defective. I've made new ones, uploaded them this evening, and then deleted the originals. When you `pip install fiona==1.8.8` and see a download of a Fiona-1.8.8-1...whl file, this is intended and all is well. There are no changes to the Python modules themselves.

--
Sean Gillies


Re: Fiona 1.8.8

Sean Gillies
 

Oops, I forgot to mention that the binary wheels on PyPI for 1.8.8 now include GDAL 2.4.2 and are lighter. We're down to about 85 MB extracted size, including GDAL and PROJ data files.

On Wed, Sep 25, 2019 at 10:06 AM Sean Gillies via Groups.Io <sean.gillies=gmail.com@groups.io> wrote:
Hi all,

Fiona 1.8.8 is on PyPI now. Please see the change log for notes about the bug fixes and future changes in 1.9.0 and 2.0.0.


This is likely to be one of the the very last 1.8.x releases.

--
Sean Gillies



--
Sean Gillies


Fiona 1.8.8

Sean Gillies
 

Hi all,

Fiona 1.8.8 is on PyPI now. Please see the change log for notes about the bug fixes and future changes in 1.9.0 and 2.0.0.


This is likely to be one of the the very last 1.8.x releases.

--
Sean Gillies


Re: How would I use fiona to detect if a GeoPackage layer is a raster?

Sean Gillies
 

Hi,

On Tue, Jul 2, 2019 at 6:25 PM <joe.huang@...> wrote:
Specifically, once I have opened the file and am accessing the data with fiona.Collection

Good question. To be honest, I don't know. I rarely see geopackage files at work and never see ones that combine vectors and rasters, so I haven't explored the situation. What happens if you try to open a raster layer with fiona?

--
Sean Gillies


How would I use fiona to detect if a GeoPackage layer is a raster?

joe.huang@...
 

Specifically, once I have opened the file and am accessing the data with fiona.Collection


Re: When Fiona opens a directory of shapefiles, does it detect if a file is missing? If so, how do I catch the error?

Sean Gillies
 

GDAL, the library that fiona uses, can succeed in the absence of some shapefile members and won't necessarily report their absence. You'll need to validate the data using other methods.

On Wed, Jun 19, 2019 at 7:39 PM <joe.huang@...> wrote:
My application requires that uploaded shapefiles require .shp, .dbf, .shx, and .prj. Is there a way that fiona.open() checks for that?



--
Sean Gillies


When Fiona opens a directory of shapefiles, does it detect if a file is missing? If so, how do I catch the error?

joe.huang@...
 

My application requires that uploaded shapefiles require .shp, .dbf, .shx, and .prj. Is there a way that fiona.open() checks for that?


Re: How do I use fiona.open() to write to a file-like object like StringIO or BytesIO?

joe.huang@...
 

Your example did the trick. Thank you so much!

By the way, thank you so much for being so quick and responsive! You are an awesome member of the open source community.


Re: How do I use fiona.open() to write to a file-like object like StringIO or BytesIO?

Sean Gillies
 

Hi,

The project has an example for you in the test suite: https://github.com/Toblerity/Fiona/blob/master/tests/test_memoryfile.py#L69-L71.

Are you using the latest version of fiona? Can you show us the full exception traceback so we can see what line it's coming from?


On Tue, Jun 18, 2019 at 4:23 PM <joe.huang@...> wrote:
I try passing the object directly into fiona.open(), but it throws a type error: string argument expected, got 'bytes'



--
Sean Gillies


How do I use fiona.open() to write to a file-like object like StringIO or BytesIO?

joe.huang@...
 

I try passing the object directly into fiona.open(), but it throws a type error: string argument expected, got 'bytes'


Re: In fiona.open(), can I pass a namedtuple into schema, rather than a dict?

Sean Gillies
 

No, namedtuples aren't supported. fiona.open() only accepts a dict or ordered dict. We're switching from dicts to custom classes in several other places in Fiona, but haven't made plans to do so for the schema.

On Thu, Jun 13, 2019 at 2:32 PM <joe.huang@...> wrote:
Named schemas are a little more lightweight and seem well-suited to the purpose of the schema dict.



--
Sean Gillies


In fiona.open(), can I pass a namedtuple into schema, rather than a dict?

joe.huang@...
 

Named schemas are a little more lightweight and seem well-suited to the purpose of the schema dict.


Re: How to use fiona.open to overwrite existing GPKG file?

Sean Gillies
 

Because a geopackage file can contain multiple layers, opening a layer in "w" mode does not overwrite the entire .gpkg file. If you want to overwrite a layer, you need to pass "overwrite=True" to fiona.open(). For example:

>>> with fiona.open("/tmp/coutwildrnp.gpkg", "w", driver="GPKG", crs=crs, schema=schema, layer="coutwildrnp", overwrite=True) as col:
...     col.write(feat)
...
>>> with fiona.open("/tmp/coutwildrnp.gpkg", layer="coutwildrnp") as col:
...     len(col)
...
1



On Thu, Jun 6, 2019 at 7:05 AM <joe.huang@...> wrote:
I'm trying to overwrite a temporary file with fiona,open, but it throws an error. Is there a way to get around this?



--
Sean Gillies


How to use fiona.open to overwrite existing GPKG file?

joe.huang@...
 

I'm trying to overwrite a temporary file with fiona,open, but it throws an error. Is there a way to get around this?

81 - 100 of 107