Date   

fiona.open returns empty collection array when using GPX files with tracks but don't include UTF-8 encoding

tlstratton@...
 

Hi,
  I'm new to Fiona and geospatial in general but so far I have been able to open 90% of GPX files and use transform_geom to get geojson formatted and returned from an endpoint. My issue is that occasionally certain files seem to open correctly but contain no data but I know the GPX file has data. I can't figure out what the issue is... these files both open fine in qgis so I know they are somewhat valid. Any ideas?

I've attached two examples.
Here's what I use to open the files:
fiona.open(file, layer='tracks') ... I also tried: fiona.open(file, layer='track_points')


Re: delete layer from geopackage?

Ari Meyer
 

Hi Sean,

I wasn't sure if you meant that this function was not currently working properly, or if it was just missing from the documentation.  In any case, I am able to execute it, but it fails:

fiona.remove('C:/SiteSelection/Senegal_test.gpkg', driver='GPKG', layer='AOI_PnLQzVP6')

Traceback (most recent call last):

  File "C:\OSGeo4W\apps\Python39\lib\code.py", line 90, in runcode

    exec(code, self.locals)

  File "<input>", line 1, in <module>

  File "C:\OSGeo4W\apps\Python39\lib\site-packages\fiona\__init__.py", line 310, in remove

    _remove_layer(path, layer, driver)

  File "fiona/ogrext.pyx", line 1640, in fiona.ogrext._remove_layer

fiona.errors.DatasetDeleteError: Failed to remove layer "AOI_PnLQzVP6" from datasource: C:/SiteSelection/Senegal_test.gpkg

I'm using QGIS LTR, which is bundling fiona 1.8.20.

FYI, if I try to run it from outside QGIS, I get the same error but more info:

ERROR 1: bad result for PRAGMA foreign_key_check, got 1 rows, expected 0

 

Do I need to anything particular prior to calling the function?

Thanks,
Ari


Re: delete layer from geopackage?

Ari Meyer
 

That's great, Sean -- thanks!

On Wed, Aug 11, 2021 at 7:04 PM Sean Gillies <sean.gillies@...> wrote:
Hi Ari,

On Tue, Jul 27, 2021 at 4:49 PM Ari Meyer <ari.meyer@...> wrote:
Thank you, Loic.  I should have said that I wasn't able to open my geopackage in write mode without having to specify a schema -- I don't want to overwrite it.  I am able to open it in append mode as you're doing, though.  In any case, I was really hoping I could open one in a "read/write" mode whereby I could examine it and make modifications, but I guess that's not possible (as the docs say: The in situ “update” mode of OGR is quite format dependent and is therefore not supported by Fiona.).  It looks like if I want to do something like that I have to open it in read mode, parse it, then write to a new geopackage.  Is that correct?

In any case, my main question is about deleting layers from a geopackage.  Does anyone know a way to do this without hand-coding a bunch of SQL DELETEs?
Ari

Thanks for the question. It turns out that fiona's API for removing a dataset or layer hasn't surfaced in our documentation due to an oversight. Until I fix that, please see


--
Sean Gillies


Re: delete layer from geopackage?

Sean Gillies
 

Hi Ari,

On Tue, Jul 27, 2021 at 4:49 PM Ari Meyer <ari.meyer@...> wrote:
Thank you, Loic.  I should have said that I wasn't able to open my geopackage in write mode without having to specify a schema -- I don't want to overwrite it.  I am able to open it in append mode as you're doing, though.  In any case, I was really hoping I could open one in a "read/write" mode whereby I could examine it and make modifications, but I guess that's not possible (as the docs say: The in situ “update” mode of OGR is quite format dependent and is therefore not supported by Fiona.).  It looks like if I want to do something like that I have to open it in read mode, parse it, then write to a new geopackage.  Is that correct?

In any case, my main question is about deleting layers from a geopackage.  Does anyone know a way to do this without hand-coding a bunch of SQL DELETEs?
Ari

Thanks for the question. It turns out that fiona's API for removing a dataset or layer hasn't surfaced in our documentation due to an oversight. Until I fix that, please see


--
Sean Gillies


Re: delete layer from geopackage?

Ari Meyer
 

Thank you, Loic.  I should have said that I wasn't able to open my geopackage in write mode without having to specify a schema -- I don't want to overwrite it.  I am able to open it in append mode as you're doing, though.  In any case, I was really hoping I could open one in a "read/write" mode whereby I could examine it and make modifications, but I guess that's not possible (as the docs say: The in situ “update” mode of OGR is quite format dependent and is therefore not supported by Fiona.).  It looks like if I want to do something like that I have to open it in read mode, parse it, then write to a new geopackage.  Is that correct?

In any case, my main question is about deleting layers from a geopackage.  Does anyone know a way to do this without hand-coding a bunch of SQL DELETEs?
Ari

On Tue, Jul 27, 2021 at 3:58 AM Loïc Dutrieux <loic.dutrieux@...> wrote:
I don't know about deleting a layer, but opening an existing geopackage layer in write or append mode does work. See example:

import string
import tempfile, os

import fiona
from fiona.crs import from_epsg
from shapely.geometry import Point, mapping


gpkg = os.path.join(tempfile.gettempdir(), 'fiona_wa_test.gpkg')
sc0 = [Point(1,2), Point(3,4), Point(5,6)]
fc0 = [{'geometry': mapping(s),
       'properties': {'letter': string.ascii_lowercase[idx]}}
     for idx,s in enumerate(sc0)]
sc1 = [Point(7,8), Point(9,10), Point(11,12)]
fc1 = [{'geometry': mapping(s),
       'properties': {'letter': string.ascii_lowercase[idx]}}
     for idx,s in enumerate(sc1)]
schema = {'geometry': 'Point',
          'properties': {'letter': 'str'}}
# Initial creation of geopackage file and layer + write fc0
with fiona.open(gpkg, 'w', driver='GPKG', schema=schema, crs=from_epsg(4326),
                layer='test') as dst:
    dst.writerecords(fc0)
# Append fc1 to existing layer
with fiona.open(gpkg, 'a', driver='GPKG', layer='test') as dst:
    dst.writerecords(fc1)

________________________________________
From: main@fiona.groups.io <main@fiona.groups.io> on behalf of Ari Meyer <ari.meyer@...>
Sent: 26 July 2021 23:33:38
To: main@fiona.groups.io
Subject: [fiona] delete layer from geopackage?

It appears that Fiona doesn't expose an API to delete a layer -- is that correct?  It seems there is no option to open a geopackage in read/write mode, and it won't even let me open an existing geopackage in write mode.  From the docs: The in situ “update” mode of OGR is quite format dependent and is therefore not supported by Fiona.  https://fiona.readthedocs.io/en/stable/manual.html#writing-vector-data<https://urldefense.com/v3/__https://fiona.readthedocs.io/en/stable/manual.html*writing-vector-data__;Iw!!DOxrgLBm!UrEkP-JHQ-cFu4XmX_M6HcSiXvZQwNKGRNK6Q_V4yE3sCWM2M1FdArQ6J1Vkn3hxrkKUTzM$>

As expected, I had similar problems trying this with OGR/GDAL.  It would seem that this is a common enough operation that there should be an API that can do this.  Right now, we have to execute a bunch of SQL DELETE calls, which is ugly and potentially error-prone.  Some have suggested a convoluted use of Geopandas: https://stackoverflow.com/questions/49291740/delete-rows-if-there-are-null-values-in-a-specific-column-in-pandas-dataframe/49291779#49291779<https://urldefense.com/v3/__https://stackoverflow.com/questions/49291740/delete-rows-if-there-are-null-values-in-a-specific-column-in-pandas-dataframe/49291779*49291779__;Iw!!DOxrgLBm!UrEkP-JHQ-cFu4XmX_M6HcSiXvZQwNKGRNK6Q_V4yE3sCWM2M1FdArQ6J1Vkn3hxXWmSdF0$> , but this is likewise not clean, IMHO.

Any suggestions?







Re: delete layer from geopackage?

Loïc Dutrieux
 

I don't know about deleting a layer, but opening an existing geopackage layer in write or append mode does work. See example:

import string
import tempfile, os

import fiona
from fiona.crs import from_epsg
from shapely.geometry import Point, mapping


gpkg = os.path.join(tempfile.gettempdir(), 'fiona_wa_test.gpkg')
sc0 = [Point(1,2), Point(3,4), Point(5,6)]
fc0 = [{'geometry': mapping(s),
'properties': {'letter': string.ascii_lowercase[idx]}}
for idx,s in enumerate(sc0)]
sc1 = [Point(7,8), Point(9,10), Point(11,12)]
fc1 = [{'geometry': mapping(s),
'properties': {'letter': string.ascii_lowercase[idx]}}
for idx,s in enumerate(sc1)]
schema = {'geometry': 'Point',
'properties': {'letter': 'str'}}
# Initial creation of geopackage file and layer + write fc0
with fiona.open(gpkg, 'w', driver='GPKG', schema=schema, crs=from_epsg(4326),
layer='test') as dst:
dst.writerecords(fc0)
# Append fc1 to existing layer
with fiona.open(gpkg, 'a', driver='GPKG', layer='test') as dst:
dst.writerecords(fc1)

________________________________________
From: main@fiona.groups.io <main@fiona.groups.io> on behalf of Ari Meyer <ari.meyer@gmail.com>
Sent: 26 July 2021 23:33:38
To: main@fiona.groups.io
Subject: [fiona] delete layer from geopackage?

It appears that Fiona doesn't expose an API to delete a layer -- is that correct? It seems there is no option to open a geopackage in read/write mode, and it won't even let me open an existing geopackage in write mode. From the docs: The in situ “update” mode of OGR is quite format dependent and is therefore not supported by Fiona. https://fiona.readthedocs.io/en/stable/manual.html#writing-vector-data<https://urldefense.com/v3/__https://fiona.readthedocs.io/en/stable/manual.html*writing-vector-data__;Iw!!DOxrgLBm!UrEkP-JHQ-cFu4XmX_M6HcSiXvZQwNKGRNK6Q_V4yE3sCWM2M1FdArQ6J1Vkn3hxrkKUTzM$>

As expected, I had similar problems trying this with OGR/GDAL. It would seem that this is a common enough operation that there should be an API that can do this. Right now, we have to execute a bunch of SQL DELETE calls, which is ugly and potentially error-prone. Some have suggested a convoluted use of Geopandas: https://stackoverflow.com/questions/49291740/delete-rows-if-there-are-null-values-in-a-specific-column-in-pandas-dataframe/49291779#49291779<https://urldefense.com/v3/__https://stackoverflow.com/questions/49291740/delete-rows-if-there-are-null-values-in-a-specific-column-in-pandas-dataframe/49291779*49291779__;Iw!!DOxrgLBm!UrEkP-JHQ-cFu4XmX_M6HcSiXvZQwNKGRNK6Q_V4yE3sCWM2M1FdArQ6J1Vkn3hxXWmSdF0$> , but this is likewise not clean, IMHO.

Any suggestions?


delete layer from geopackage?

Ari Meyer
 

It appears that Fiona doesn't expose an API to delete a layer -- is that correct?  It seems there is no option to open a geopackage in read/write mode, and it won't even let me open an existing geopackage in write mode.  From the docs: The in situ “update” mode of OGR is quite format dependent and is therefore not supported by Fiona.  https://fiona.readthedocs.io/en/stable/manual.html#writing-vector-data

As expected, I had similar problems trying this with OGR/GDAL.  It would seem that this is a common enough operation that there should be an API that can do this.  Right now, we have to execute a bunch of SQL DELETE calls, which is ugly and potentially error-prone.  Some have suggested a convoluted use of Geopandas: https://stackoverflow.com/questions/49291740/delete-rows-if-there-are-null-values-in-a-specific-column-in-pandas-dataframe/49291779#49291779 , but this is likewise not clean, IMHO.

Any suggestions?


Re: fiona SQL filter

Loïc Dutrieux
 

Thanks Alan, you're right. I wrongly assumed that this was an old or at least already existing feature since it is in the online doc.

Cheers,
Loïc
________________________________________
From: main@fiona.groups.io <main@fiona.groups.io> on behalf of Alan Snow <alansnow21@gmail.com>
Sent: 03 June 2021 20:05:29
To: main@fiona.groups.io
Subject: Re: [fiona] fiona SQL filter

I think the feature you want is slated for 2.0: https://github.com/Toblerity/Fiona/pull/961<https://urldefense.com/v3/__https://github.com/Toblerity/Fiona/pull/961__;!!DOxrgLBm!UB5tJPQHin0tXeTB6cruOVmD3nIa-Dq8tSRv6xve6RsrgL5eedhcf0yBmtwj9Z5O8Cik8ns$>


Re: fiona SQL filter

Alan Snow
 

I think the feature you want is slated for 2.0: https://github.com/Toblerity/Fiona/pull/961


fiona SQL filter

Loïc Dutrieux
 

Hi everyone,

I'm trying to do some attribute filtering of a fiona feature collection using the where clause but I cannot find out the right syntax. Here's what I've tried.

Thanks for any help,
Kind regards,
Loïc


import string
import tempfile, os

import fiona
from fiona.crs import from_epsg
from shapely.geometry import Point, mapping

# Generate test data and save to tmp geopackage
gpkg = os.path.join(tempfile.gettempdir(), 'fiona_sql_test.gpkg')
sc = [Point(1,2), Point(3,4), Point(5,6)]
fc = [{'geometry': mapping(s),
'properties': {'letter': string.ascii_lowercase[idx]}}
for idx,s in enumerate(sc)]
schema = {'geometry': 'Point',
'properties': {'letter': 'str'}}
with fiona.open(gpkg, 'w', driver='GPKG', schema=schema, crs=from_epsg(4326),
layer='test') as dst:
dst.writerecords(fc)

# Filter using OGR SQL dialect
with fiona.open(gpkg, layer='test') as src:
fc = src.filter(where='"letter" = "a"')
for feat in fc:
print(feat)


Fiona 1.8.20

Sean Gillies
 

Hi all,

Fiona 1.8.20 is on PyPI today. No changes other than a relaxation of the constraint on click version. Any version of click >= 4.0 is allowed now. The wheels on PyPI include library upgrades: GDAL 3.3.0 and GEOS 3.9.1.

--
Sean Gillies


Re: Cannot install Fiona on Mac OS, GDAL installed via homebrew

jackjomarron@...
 

Thank you for your help!

On Sun, Apr 25, 2021 at 11:35 AM <camille.modeste@...> wrote:
Hi, 

I got the same issue, with pipenv, running on Python 3.9.4. Setting Python to 3.9.1 solved this issue.



--
Jack Jomarron

   


Re: Cannot install Fiona on Mac OS, GDAL installed via homebrew

camille.modeste@...
 

Hi, 

I got the same issue, with pipenv, running on Python 3.9.4. Setting Python to 3.9.1 solved this issue.


AttributeError on import; module 'fiona' has no attribute '_loading'

jackjomarron@...
 

I am trying to map on Jupyter notebook using geopandas. I am running into some issues importing geopandas through fiona. I have tried importing fiona directly as well and still receive the same error.

Along with importing fiona directly, I have tried uninstalling and reinstalling fiona and calling the _loading attribute directly.

I have checked on conda list and have gdal 3.2.2 and fiona 1.8.19 both on the conda-forge channel.

Thanks for any help!


Re: FIona 1.1.18, GDAL 3.2.1

René Buffat
 


If you install your own proj and gdal, then you probably don't want to use the wheel version of Fiona, as the wheels contain their own version of gdal. 
With the following command you should be able to compile Fiona from source using your installed gdal:

pip install Fiona --no-binary Fiona

 

Regarding the error message: gcs.csv was replaced by a sqlite file in gdal 3.0. The current Fiona wheels include still gdal 2.4. I assume you are affected by https://github.com/Toblerity/Fiona/issues/1007

where the old gdal from the wheel finds the GDAL_DATA directory of the new gdal installation. 

 

lg rene

 

 


FIona 1.1.18, GDAL 3.2.1

David Ignjić
 

I use docker file (later) where i install fiona via pip when i make 
import fiona

I get error:
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.

Do i need compile fiona ?

Thanks 
                David





Docker file:
FROM python:3.8-slim-buster

ENV DEBIAN_FRONTEND noninteractive
ENV POETRY_VERSION 1.1.4
ENV PROJ_VERSION 7.2.1
ENV GDAL_VERSION 3.2.1

RUN apt-get update -yqq \
&& apt-get install -yqq build-essential wget pkg-config file libsqlite3-dev sqlite3 libtiff-dev netcat vim procps

# PROJ
RUN wget https://download.osgeo.org/proj/proj-${PROJ_VERSION}.tar.gz \
&& tar xzf proj-${PROJ_VERSION}.tar.gz \
&& cd proj-${PROJ_VERSION} \
&& ./configure --prefix=/usr --sysconfdir=/etc --without-curl \
&& make --quiet -j4 \
&& make --quiet install \
&& make --quiet clean \
&& cd .. \
&& rm -rf proj-${PROJ_VERSION} \
&& rm -f proj-${PROJ_VERSION}.tar.gz


# GDAL

RUN wget http://download.osgeo.org/gdal/${GDAL_VERSION}/gdal-${GDAL_VERSION}.tar.gz \
&& tar xzf gdal-${GDAL_VERSION}.tar.gz \
&& cd gdal-${GDAL_VERSION} \
&& ./configure --disable-static --prefix=/usr --sysconfdir=/etc --with-libkml \
&& make --quiet -j4\
&& make --quiet install \
&& make --quiet clean \
&& cd .. \
&& rm -rf gdal-${GDAL_VERSION} \
&& rm -f gdal-${GDAL_VERSION}.tar.gz


Re: Attempting to find version of GDAL installed

Orry Messer
 

Yeah, I believe this is what has happened. On the CI/CD server, a whl was used, and locally it's using the source .tar.gz.

Thanks again.


Re: Attempting to find version of GDAL installed

Sean Gillies
 

We provide wheel distributions for only a small group of platform tags and Python versions. See https://pypi.org/project/Fiona/1.8.18/#files. I suspect your CI/CD server's pip program finds a compatible wheel, but your dev machine cannot. In that case, pip on your dev machine will take the source distribution, the .tar.gz file.


On Mon, Mar 8, 2021, 12:44 AM Orry Messer <orry@...> wrote:
Thanks Sean!

Well, it appears that the libgdal shared library which fiona is linking resides in /usr/local/lib/python3.6/site-packages/fiona/.libs, rather than /usr/lib or /usr/local/lib - which leads me to believe that fiona has obtained its GDAL dependencies from PyPI. 

What I am struggling with, is that locally, on my dev machine, fiona does not seem to be getting GDAL dependencies from PyPI, where as on my CI/CD server, it is getting its dependencies and so the result of "pip install fiona" works. Do you have any idea why this may be the case?

- Orry


Re: Attempting to find version of GDAL installed

René Buffat
 

Not sure if related, but there seems to be no macos wheel for python 3.9 on pypi: https://pypi.org/project/Fiona/#files


Re: Attempting to find version of GDAL installed

Orry Messer
 

Thanks Sean!

Well, it appears that the libgdal shared library which fiona is linking resides in /usr/local/lib/python3.6/site-packages/fiona/.libs, rather than /usr/lib or /usr/local/lib - which leads me to believe that fiona has obtained its GDAL dependencies from PyPI. 

What I am struggling with, is that locally, on my dev machine, fiona does not seem to be getting GDAL dependencies from PyPI, where as on my CI/CD server, it is getting its dependencies and so the result of "pip install fiona" works. Do you have any idea why this may be the case?

- Orry

1 - 20 of 111