Date   

Re: Unable to open .zip in Docker container

davejohansen@...
 

I'm using fiona 1.8.13 in both case, and for GDAL, on my Mac, it's 2.4.2, in the debian container it's 2.4.0 and in the Alpine container it's 3.0.3.


On Wed, Feb 19, 2020 at 12:18 AM <hannes-fiona.groups.io@...> wrote:
Hi Dave,

are you using the same versions of Fiona and GDAL?

Cheers, Hannes

On Tue, 18 Feb 2020 16:25:57 -0800
davejohansen@... wrote:

> I'm trying to load a .gdb.zip file in a Docker container using
> `fiona.listlayers()` but it says that it's not a supported format
> with the following exception:
>
> ```
>
> File \"fiona/_shim.pyx\", line 83, in fiona._shim.gdal_open_vector
>
> File \"fiona/_err.pyx\", line 270, in fiona._err.exc_wrap_pointer
>
> fiona._err.CPLE_OpenFailedError: '/tmp/test.gdb.zip' not recognized
> as a supported file format.
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>
> File \"/usr/src/app/numetric/connectrunner/shp_file/shp_file.py\",
> line 34, in _listlayers
>
> for layer in fiona.listlayers(filename):
>
> File
> \"/usr/src/app/numetric/connectrunner/venv/lib/python3.8/site-packages/fiona/env.py\",
> line 407, in wrapper
>
> return f(*args, **kwargs)
>
> File
> \"/usr/src/app/numetric/connectrunner/venv/lib/python3.8/site-packages/fiona/__init__.py\",
> line 344, in listlayers
>
> return _listlayers(vsi_path(pobj))
>
> File \"fiona/ogrext.pyx\", line 1512, in fiona.ogrext._listlayers
>
> File \"fiona/_shim.pyx\", line 90, in fiona._shim.gdal_open_vector
> ```
>
> The same code works fine on my Mac with the same file, but doesn't in
> the `3.8.1-slim-buster` and `3.8.1-alpine3.11` images. Any ideas on
> why and what I can do to fix this?
>
> Thanks,
>
> Dave
>
>
>




Re: Unable to open .zip in Docker container

hannes-fiona.groups.io@...
 

Hi Dave,

are you using the same versions of Fiona and GDAL?

Cheers, Hannes

On Tue, 18 Feb 2020 16:25:57 -0800
davejohansen@gmail.com wrote:

I'm trying to load a .gdb.zip file in a Docker container using
`fiona.listlayers()` but it says that it's not a supported format
with the following exception:

```

File \"fiona/_shim.pyx\", line 83, in fiona._shim.gdal_open_vector

File \"fiona/_err.pyx\", line 270, in fiona._err.exc_wrap_pointer

fiona._err.CPLE_OpenFailedError: '/tmp/test.gdb.zip' not recognized
as a supported file format.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File \"/usr/src/app/numetric/connectrunner/shp_file/shp_file.py\",
line 34, in _listlayers

for layer in fiona.listlayers(filename):

File
\"/usr/src/app/numetric/connectrunner/venv/lib/python3.8/site-packages/fiona/env.py\",
line 407, in wrapper

return f(*args, **kwargs)

File
\"/usr/src/app/numetric/connectrunner/venv/lib/python3.8/site-packages/fiona/__init__.py\",
line 344, in listlayers

return _listlayers(vsi_path(pobj))

File \"fiona/ogrext.pyx\", line 1512, in fiona.ogrext._listlayers

File \"fiona/_shim.pyx\", line 90, in fiona._shim.gdal_open_vector
```

The same code works fine on my Mac with the same file, but doesn't in
the `3.8.1-slim-buster` and `3.8.1-alpine3.11` images. Any ideas on
why and what I can do to fix this?

Thanks,

Dave



Unable to open .zip in Docker container

davejohansen@...
 

I'm trying to load a .gdb.zip file in a Docker container using `fiona.listlayers()` but it says that it's not a supported format with the following exception:

```

File \"fiona/_shim.pyx\", line 83, in fiona._shim.gdal_open_vector

  File \"fiona/_err.pyx\", line 270, in fiona._err.exc_wrap_pointer

fiona._err.CPLE_OpenFailedError: '/tmp/test.gdb.zip' not recognized as a supported file format.

 

During handling of the above exception, another exception occurred:

 

Traceback (most recent call last):

  File \"/usr/src/app/numetric/connectrunner/shp_file/shp_file.py\", line 34, in _listlayers

    for layer in fiona.listlayers(filename):

  File \"/usr/src/app/numetric/connectrunner/venv/lib/python3.8/site-packages/fiona/env.py\", line 407, in wrapper

    return f(*args, **kwargs)

  File \"/usr/src/app/numetric/connectrunner/venv/lib/python3.8/site-packages/fiona/__init__.py\", line 344, in listlayers

    return _listlayers(vsi_path(pobj))

  File \"fiona/ogrext.pyx\", line 1512, in fiona.ogrext._listlayers

 

  File \"fiona/_shim.pyx\", line 90, in fiona._shim.gdal_open_vector
```

The same code works fine on my Mac with the same file, but doesn't in the `3.8.1-slim-buster` and `3.8.1-alpine3.11` images. Any ideas on why and what I can do to fix this?

Thanks,

Dave


Re: supplying SHAPE_RESTORE_SHX=YES to fiona.open

René Buffat
 

You should be able to pass configuration options to GDAL using fiona.ENV: https://fiona.readthedocs.io/en/stable/manual.html#ogr-configuration-options

According to Wikipedia, at least the shp, shx and dbf files are mandatory: https://en.wikipedia.org/wiki/Shapefile#Shapefile_shape_index_format_(.shx)

GDAL, respectively Fiona is able to read dxf files directly: https://gdal.org/drivers/vector/dxf.html As DXF is not a standard GIS file format, it's probably best to try out what works better.

lg rene


supplying SHAPE_RESTORE_SHX=YES to fiona.open

Ricky Teachey
 

Hello: I hope this is the right place to ask a question.

I am trying to figure out how to open a .shp while without all the other accompanying files. I THINK want to do this because I believe I don't care about the data in those other files. However I'm also a total beginner in usage of shape files so please let me know if I am wrong about what it is I believe.

My belief is that the geometric information- basically, the XY coordinates- of the shapes I intend to plot (using plotly.express.choropleth) and turn into a little app for my own use is contained in the .shp file. These .shp files are being created for me in Autocad by a coworker; they are not actually maps of any kind, just a bunch of shapes, and .shp is a much more convenient way to save them than .dxf.

I understand that most of the time, you will pass shape files around with other files, such as .shx, and Autocad does create these files. However it would be easier to not worry about these other files for my purposes.

It appears it is possible to do what I want to do by supplying the option of SHAPE_RESTORE_SHX=YES to GDAL. However, I'm trying to do this using a python script with fiona, and not using the CLI.

Assuming what I am wanting to do makes some kind of sense, is there a way to supply the SHAPE_RESTORE_SHX=YES directive to fiona.open so that I can open a .shp file without the other files? I tried just doing this:
fiona.open(pathlib.Path(shp), SHAPE_RESTORE_SHX="YES")
But it doesn't seem to work.

details:
Python 3.8.1
Windows 10
Fiona 1.8.13
GDAL 3.0.4
Installed using the wheels at Christoph Gohlke's web page

---
Ricky.

"I've never met a Kentucky man who wasn't either thinking about going home or actually going home." - Happy Chandler


Fiona 1.8.13 and py38 manylinux1 and macosx wheels

Sean Gillies
 

Hi all,

You may not notice the changes in 1.8.13 unless you're using poetry to install Fiona, but we do have 64-bit manylinux1 and macosx wheels for Python 3.8 for the first time. I hope you enjoy them and thank you for your patience!

--
Sean Gillies


Re: Scaling Geometry to Match Image File

katsonandrew3.5@...
 

Update:

I saw the response to getting it in WGS84 and I will try that in a bit. But I think the major issue is that it actually does not include antarctic and cuts off greenland as shown in this representation (http://www.luminocity3d.org/WorldPopDen/#3/45.89/-7.73)


Re: Scaling Geometry to Match Image File

katsonandrew3.5@...
 

In case this helps. Here is how I did scaling it to get more accurate numbers originally before I switched but this was even flawed:

world_total_bounds = world_boundaries.total_bounds

population_image_total_bounds = world_pop_image.bounds

x_scale = (population_image_total_bounds.right - population_image_total_bounds.left) / (world_total_bounds[MAX_X] - world_total_bounds[MIN_X])
y_scale = (population_image_total_bounds.top - population_image_total_bounds.bottom) / (world_total_bounds[MAX_Y] - world_total_bounds[MIN_Y])

world_boundaries['geometry'] = world_boundaries['geometry'].apply(lambda geometry: shapely.ops.transform(lambda x, y, z=None: (x * x_scale, y * y_scale), geometry))


Re: Scaling Geometry to Match Image File

katsonandrew3.5@...
 

I do not know if this will be helpful at all but this is how I am doing my masks and population:


population_image_slice_arr, new_image_transform = rasterio.mask.mask(world_pop_image, [polygon_for_some_country], crop=True, nodata=0)

population_image_slice_arr[population_image_slice_arr < 0] = 0

total_population = population_image_slice_arr.sum()


I know this is incorrect because when i did manual scaling using the bounds of the image and shapely.ops.transform the population for say India would be 800 million when it should be 1.3 billion but then when I changed it to use the affine matrix the math came out to 0.0 population.


Re: Scaling Geometry to Match Image File

katsonandrew3.5@...
 

I did not finish my question. I have typed the rest.

Here is how I read in the population file:

world_pop_image = rasterio.open(path_to_image, nodata=0)

Here is how I read the boundaries file:

world_boundaries = gpd.read_file(path_to_boundaries)

Here is how I do my reprojection (the raster is in World Mollweide which is not explicitly supported so I found the below workaround):

world_boundaries.to_crs('+proj=eck4 +lon_0=0 +x_0=0 +y_0=0 +datum=WGS84 +units=m +no_defs')

Here is how I do my scaling:

from shapely.affinity import affine_transformation

population_image_affine = world_pop_image.transform

shapely_affine_repr = [population_image_affine.a, population_image_affine.b, population_image_affine.d, population_image_affine.e, population_image_affine.xoff, population_image_affine.yoff]

world_boundaries['geometry'] = world_boundaries['geometry].apply(lambda geometry: affine_transformation(geometry, shapely_affine_repr)


Scaling Geometry to Match Image File

katsonandrew3.5@...
 

Hi,

I know this is not a fiona specific question but I figure since this kind of covers a variety of modules written by the same people (or at least understood) that I might get an answer here.

I am attempting to do a mask over a population image file using geometries but they are in different projections and different scales. Here are my modules below.

I am using this set of land boundaries: https://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/ne_10m_admin_0_boundary_lines_land.zip
and this population image file: https://ghsl.jrc.ec.europa.eu/download.php?ds=pop (with the 250 m resolution)

Here is how I read in the population file:

world_pop_image =

Here is how I do my reprojection:


Below are all my python packages

Python 3.6
with
Click 7
Fiona 1.8.11
Geometry 0.0.23
Pillow 6.2.1
PyContracts 1.8.12
PyGeometry 1.5.6
Rtree 0.8.3
Shapely 1.6.4.post2
affine 2.3.0
atomicwrites 1.3.0
attrs 19.3.0
certifi 2019.9.11
chardet 3.0.4
click-plugins 1.1.1
cligj 0.5.0
coverage 4.5.4
cupy 6.5.0
cycler 0.10.0
decorator 4.4.1
descartes 1.1.0
fastrlock 0.4
future 0.18.2
geopandas 0.6.1
h5py 2.10.0
idna 2.8
imageio 2.6.1
importlib-metadata 0.23
kiwisolver 1.1.0
matplotlib 3.1.1
more-itertools 7.2.0
munch 2.5.0
numpy 1.17.4
packaging 19.2
pandas 0.25.0
pip 19.3.1
pluggy 0.13.0
psutil 5.6.5
py 1.8.0
pyparsing 2.4.5
pyproj 2.4.1
pytest 5.2.1
pytest-cov 2.8.1
python-Levenshtein 0.12.0
python-dateutil 2.8.1
pytz 2019.3
rasterio 1.0.25
requests 2.22.0
scipy 1.3.1
setuptools 41.6.0
six 1.13.0
snuggs 1.4.7
urllib3 1.25.7
wcwidth 0.1.7
zipp 0.6.0


Re: Where to Ask Questions

katsonandrew3.5@...
 

Great thank you. I will put together some information!


Re: Where to Ask Questions

Sean Gillies
 

Hi,

This discussion group is the right place to ask for help. Make sure to provide details when you describe your problem and the steps you've taken to try to solve it. Python version, operating system, Fiona version, and the source of your software (pip? conda? elsewhere?) are often critical clues.


On Mon, Nov 18, 2019 at 6:34 AM <katsonandrew3.5@...> wrote:
Hi I have been working with shapely and fiona for a few months now. I have an issue that I found mentioned on the github issues but the solution to it does not apply to mine. I was wondering where I could ask in case anyone could help me out. Thank you!

This is the issue I am referencing https://github.com/Toblerity/Fiona/issues/645



--
Sean Gillies


Where to Ask Questions

katsonandrew3.5@...
 

Hi I have been working with shapely and fiona for a few months now. I have an issue that I found mentioned on the github issues but the solution to it does not apply to mine. I was wondering where I could ask in case anyone could help me out. Thank you!

This is the issue I am referencing https://github.com/Toblerity/Fiona/issues/645


Re: Extremely slow performance with shapefile and geopackage

Sean Gillies
 

David,

I am not a geopackage expert, but yes, it looks like your sqlite3 library lacks the rtree extension.

In the future, please provide these kind of error messages and some details about the sources of your software (conda? pip?) when you ask for help. That will help us get to the heart of the matter and prevent speculation about things like transaction sizes.


On Sun, Nov 17, 2019 at 2:33 AM David Epstein <davideps@...> wrote:
Thank you Sean and Rene. When I load only 1,000 features and save as a shapefile, it writes quickly and correctly (i.e. it is loadable in QGIS). However, I get a new error when I save to geopackage. I think I need to recompile SQLITE with RTREE support, right? Do I do this directly with a C-compiler or is this something that I can do indirectly via conda or pip? Here is the error:

>>> geodata.to_file(data_dir+"data.gpkg", driver="GPKG")
Traceback (most recent call last):
  File "fiona/_err.pyx", line 201, in fiona._err.GDALErrCtxManager.__exit__
fiona._err.CPLE_AppDefinedError: b'sqlite3_exec(CREATE VIRTUAL TABLE "rtree_data_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: no such module: rtree'
Exception ignored in: 'fiona._shim.gdal_flush_cache'
Traceback (most recent call last):
  File "fiona/_err.pyx", line 201, in fiona._err.GDALErrCtxManager.__exit__
fiona._err.CPLE_AppDefinedError: b'sqlite3_exec(CREATE VIRTUAL TABLE "rtree_data_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: no such module: rtree'

On Thu, Nov 14, 2019 at 7:11 PM Sean Gillies <sean.gillies@...> wrote:
René, this is all good advice.

In the past, saving to a geopackage could be especially slow because of the overhead of transactions. Every Collection write() would happen within its own transaction. Since version 1.8 of Fiona, calling writerecords() uses a default transaction size of 20,000 features (see https://fiona.readthedocs.io/en/latest/README.html#a1-2017-11-06) and is much faster.

Geopandas uses the faster method since https://github.com/geopandas/geopandas/issues/557#issuecomment-332202764. You may want to check to see if a geopandas upgrade improves your situation.

On Thu, Nov 14, 2019 at 8:33 AM René Buffat <buffat@...> wrote:
Hi David

First I would check if you have a sufficient amount of RAM available. If not, this could explain the slow performance.
If this is the case, I would recommend to read, process and write the data in batches.

Otherwise, there are a lot of parameters that can impact the performance. E.g. how complex the geometries are, how many rows you want to write, how many parallel reads and write you have to the disk, etc.

Regarding geometries problems, I'm not entirely sure what you mean. But regardless, with big datasets, it's always a good option to debug with smaller datasets (e.g. the first thousand lines) and then test if everything works. 

And fully unrelated, I would recommend to use os.path.join(datadir, "data.shp") instead of data_dir+"data.gpkg"

lg rene



--
Sean Gillies



--
Sean Gillies


Re: Extremely slow performance with shapefile and geopackage

davideps@...
 

Thank you Sean and Rene. When I load only 1,000 features and save as a shapefile, it writes quickly and correctly (i.e. it is loadable in QGIS). However, I get a new error when I save to geopackage. I think I need to recompile SQLITE with RTREE support, right? Do I do this directly with a C-compiler or is this something that I can do indirectly via conda or pip? Here is the error:

>>> geodata.to_file(data_dir+"data.gpkg", driver="GPKG")
Traceback (most recent call last):
  File "fiona/_err.pyx", line 201, in fiona._err.GDALErrCtxManager.__exit__
fiona._err.CPLE_AppDefinedError: b'sqlite3_exec(CREATE VIRTUAL TABLE "rtree_data_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: no such module: rtree'
Exception ignored in: 'fiona._shim.gdal_flush_cache'
Traceback (most recent call last):
  File "fiona/_err.pyx", line 201, in fiona._err.GDALErrCtxManager.__exit__
fiona._err.CPLE_AppDefinedError: b'sqlite3_exec(CREATE VIRTUAL TABLE "rtree_data_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: no such module: rtree'


On Thu, Nov 14, 2019 at 7:11 PM Sean Gillies <sean.gillies@...> wrote:
René, this is all good advice.

In the past, saving to a geopackage could be especially slow because of the overhead of transactions. Every Collection write() would happen within its own transaction. Since version 1.8 of Fiona, calling writerecords() uses a default transaction size of 20,000 features (see https://fiona.readthedocs.io/en/latest/README.html#a1-2017-11-06) and is much faster.

Geopandas uses the faster method since https://github.com/geopandas/geopandas/issues/557#issuecomment-332202764. You may want to check to see if a geopandas upgrade improves your situation.

On Thu, Nov 14, 2019 at 8:33 AM René Buffat <buffat@...> wrote:
Hi David

First I would check if you have a sufficient amount of RAM available. If not, this could explain the slow performance.
If this is the case, I would recommend to read, process and write the data in batches.

Otherwise, there are a lot of parameters that can impact the performance. E.g. how complex the geometries are, how many rows you want to write, how many parallel reads and write you have to the disk, etc.

Regarding geometries problems, I'm not entirely sure what you mean. But regardless, with big datasets, it's always a good option to debug with smaller datasets (e.g. the first thousand lines) and then test if everything works. 

And fully unrelated, I would recommend to use os.path.join(datadir, "data.shp") instead of data_dir+"data.gpkg"

lg rene



--
Sean Gillies


Re: Extremely slow performance with shapefile and geopackage

Sean Gillies
 

René, this is all good advice.

In the past, saving to a geopackage could be especially slow because of the overhead of transactions. Every Collection write() would happen within its own transaction. Since version 1.8 of Fiona, calling writerecords() uses a default transaction size of 20,000 features (see https://fiona.readthedocs.io/en/latest/README.html#a1-2017-11-06) and is much faster.

Geopandas uses the faster method since https://github.com/geopandas/geopandas/issues/557#issuecomment-332202764. You may want to check to see if a geopandas upgrade improves your situation.

On Thu, Nov 14, 2019 at 8:33 AM René Buffat <buffat@...> wrote:
Hi David

First I would check if you have a sufficient amount of RAM available. If not, this could explain the slow performance.
If this is the case, I would recommend to read, process and write the data in batches.

Otherwise, there are a lot of parameters that can impact the performance. E.g. how complex the geometries are, how many rows you want to write, how many parallel reads and write you have to the disk, etc.

Regarding geometries problems, I'm not entirely sure what you mean. But regardless, with big datasets, it's always a good option to debug with smaller datasets (e.g. the first thousand lines) and then test if everything works. 

And fully unrelated, I would recommend to use os.path.join(datadir, "data.shp") instead of data_dir+"data.gpkg"

lg rene



--
Sean Gillies


Re: Extremely slow performance with shapefile and geopackage

René Buffat
 

Hi David

First I would check if you have a sufficient amount of RAM available. If not, this could explain the slow performance.
If this is the case, I would recommend to read, process and write the data in batches.

Otherwise, there are a lot of parameters that can impact the performance. E.g. how complex the geometries are, how many rows you want to write, how many parallel reads and write you have to the disk, etc.

Regarding geometries problems, I'm not entirely sure what you mean. But regardless, with big datasets, it's always a good option to debug with smaller datasets (e.g. the first thousand lines) and then test if everything works. 

And fully unrelated, I would recommend to use os.path.join(datadir, "data.shp") instead of data_dir+"data.gpkg"

lg rene


Extremely slow performance with shapefile and geopackage

davideps@...
 

Hi folks,

I loaded a 5GB CSV file, turned lat/long into points, and have tried to save as shapefile and geopackage. At first, I ran into problems with datetime and tuple columns, which I've since turned to STR or dropped entirely. However, after an hour I still was not able to save. Lines like:
geodata.to_file(filename=data_dir+"data.shp"driver = 'ESRI Shapefile')
geodata.to_file(data_dir+"data.gpkg"driver="GPKG")
seem to hang with empty files written to disk. When I kill the line in the interpreter, sometimes data gets written at the last minute--sometimes not. Are there common dateframe or geometry problems I should check first? 

-david


 


Fiona 1.8.10

Sean Gillies
 

Hi all, 

1.8.10 is on PyPI now. A new Windows bug has been found and so we'll have a 1.8.11 release early tomorrow. 

61 - 80 of 104