Date   

Re: delete layer from geopackage?

Ari Meyer
 

Thank you, Loic.  I should have said that I wasn't able to open my geopackage in write mode without having to specify a schema -- I don't want to overwrite it.  I am able to open it in append mode as you're doing, though.  In any case, I was really hoping I could open one in a "read/write" mode whereby I could examine it and make modifications, but I guess that's not possible (as the docs say: The in situ “update” mode of OGR is quite format dependent and is therefore not supported by Fiona.).  It looks like if I want to do something like that I have to open it in read mode, parse it, then write to a new geopackage.  Is that correct?

In any case, my main question is about deleting layers from a geopackage.  Does anyone know a way to do this without hand-coding a bunch of SQL DELETEs?
Ari

On Tue, Jul 27, 2021 at 3:58 AM Loïc Dutrieux <loic.dutrieux@...> wrote:
I don't know about deleting a layer, but opening an existing geopackage layer in write or append mode does work. See example:

import string
import tempfile, os

import fiona
from fiona.crs import from_epsg
from shapely.geometry import Point, mapping


gpkg = os.path.join(tempfile.gettempdir(), 'fiona_wa_test.gpkg')
sc0 = [Point(1,2), Point(3,4), Point(5,6)]
fc0 = [{'geometry': mapping(s),
       'properties': {'letter': string.ascii_lowercase[idx]}}
     for idx,s in enumerate(sc0)]
sc1 = [Point(7,8), Point(9,10), Point(11,12)]
fc1 = [{'geometry': mapping(s),
       'properties': {'letter': string.ascii_lowercase[idx]}}
     for idx,s in enumerate(sc1)]
schema = {'geometry': 'Point',
          'properties': {'letter': 'str'}}
# Initial creation of geopackage file and layer + write fc0
with fiona.open(gpkg, 'w', driver='GPKG', schema=schema, crs=from_epsg(4326),
                layer='test') as dst:
    dst.writerecords(fc0)
# Append fc1 to existing layer
with fiona.open(gpkg, 'a', driver='GPKG', layer='test') as dst:
    dst.writerecords(fc1)

________________________________________
From: main@fiona.groups.io <main@fiona.groups.io> on behalf of Ari Meyer <ari.meyer@...>
Sent: 26 July 2021 23:33:38
To: main@fiona.groups.io
Subject: [fiona] delete layer from geopackage?

It appears that Fiona doesn't expose an API to delete a layer -- is that correct?  It seems there is no option to open a geopackage in read/write mode, and it won't even let me open an existing geopackage in write mode.  From the docs: The in situ “update” mode of OGR is quite format dependent and is therefore not supported by Fiona.  https://fiona.readthedocs.io/en/stable/manual.html#writing-vector-data<https://urldefense.com/v3/__https://fiona.readthedocs.io/en/stable/manual.html*writing-vector-data__;Iw!!DOxrgLBm!UrEkP-JHQ-cFu4XmX_M6HcSiXvZQwNKGRNK6Q_V4yE3sCWM2M1FdArQ6J1Vkn3hxrkKUTzM$>

As expected, I had similar problems trying this with OGR/GDAL.  It would seem that this is a common enough operation that there should be an API that can do this.  Right now, we have to execute a bunch of SQL DELETE calls, which is ugly and potentially error-prone.  Some have suggested a convoluted use of Geopandas: https://stackoverflow.com/questions/49291740/delete-rows-if-there-are-null-values-in-a-specific-column-in-pandas-dataframe/49291779#49291779<https://urldefense.com/v3/__https://stackoverflow.com/questions/49291740/delete-rows-if-there-are-null-values-in-a-specific-column-in-pandas-dataframe/49291779*49291779__;Iw!!DOxrgLBm!UrEkP-JHQ-cFu4XmX_M6HcSiXvZQwNKGRNK6Q_V4yE3sCWM2M1FdArQ6J1Vkn3hxXWmSdF0$> , but this is likewise not clean, IMHO.

Any suggestions?







Re: delete layer from geopackage?

Loïc Dutrieux
 

I don't know about deleting a layer, but opening an existing geopackage layer in write or append mode does work. See example:

import string
import tempfile, os

import fiona
from fiona.crs import from_epsg
from shapely.geometry import Point, mapping


gpkg = os.path.join(tempfile.gettempdir(), 'fiona_wa_test.gpkg')
sc0 = [Point(1,2), Point(3,4), Point(5,6)]
fc0 = [{'geometry': mapping(s),
'properties': {'letter': string.ascii_lowercase[idx]}}
for idx,s in enumerate(sc0)]
sc1 = [Point(7,8), Point(9,10), Point(11,12)]
fc1 = [{'geometry': mapping(s),
'properties': {'letter': string.ascii_lowercase[idx]}}
for idx,s in enumerate(sc1)]
schema = {'geometry': 'Point',
'properties': {'letter': 'str'}}
# Initial creation of geopackage file and layer + write fc0
with fiona.open(gpkg, 'w', driver='GPKG', schema=schema, crs=from_epsg(4326),
layer='test') as dst:
dst.writerecords(fc0)
# Append fc1 to existing layer
with fiona.open(gpkg, 'a', driver='GPKG', layer='test') as dst:
dst.writerecords(fc1)

________________________________________
From: main@fiona.groups.io <main@fiona.groups.io> on behalf of Ari Meyer <ari.meyer@gmail.com>
Sent: 26 July 2021 23:33:38
To: main@fiona.groups.io
Subject: [fiona] delete layer from geopackage?

It appears that Fiona doesn't expose an API to delete a layer -- is that correct? It seems there is no option to open a geopackage in read/write mode, and it won't even let me open an existing geopackage in write mode. From the docs: The in situ “update” mode of OGR is quite format dependent and is therefore not supported by Fiona. https://fiona.readthedocs.io/en/stable/manual.html#writing-vector-data<https://urldefense.com/v3/__https://fiona.readthedocs.io/en/stable/manual.html*writing-vector-data__;Iw!!DOxrgLBm!UrEkP-JHQ-cFu4XmX_M6HcSiXvZQwNKGRNK6Q_V4yE3sCWM2M1FdArQ6J1Vkn3hxrkKUTzM$>

As expected, I had similar problems trying this with OGR/GDAL. It would seem that this is a common enough operation that there should be an API that can do this. Right now, we have to execute a bunch of SQL DELETE calls, which is ugly and potentially error-prone. Some have suggested a convoluted use of Geopandas: https://stackoverflow.com/questions/49291740/delete-rows-if-there-are-null-values-in-a-specific-column-in-pandas-dataframe/49291779#49291779<https://urldefense.com/v3/__https://stackoverflow.com/questions/49291740/delete-rows-if-there-are-null-values-in-a-specific-column-in-pandas-dataframe/49291779*49291779__;Iw!!DOxrgLBm!UrEkP-JHQ-cFu4XmX_M6HcSiXvZQwNKGRNK6Q_V4yE3sCWM2M1FdArQ6J1Vkn3hxXWmSdF0$> , but this is likewise not clean, IMHO.

Any suggestions?


delete layer from geopackage?

Ari Meyer
 

It appears that Fiona doesn't expose an API to delete a layer -- is that correct?  It seems there is no option to open a geopackage in read/write mode, and it won't even let me open an existing geopackage in write mode.  From the docs: The in situ “update” mode of OGR is quite format dependent and is therefore not supported by Fiona.  https://fiona.readthedocs.io/en/stable/manual.html#writing-vector-data

As expected, I had similar problems trying this with OGR/GDAL.  It would seem that this is a common enough operation that there should be an API that can do this.  Right now, we have to execute a bunch of SQL DELETE calls, which is ugly and potentially error-prone.  Some have suggested a convoluted use of Geopandas: https://stackoverflow.com/questions/49291740/delete-rows-if-there-are-null-values-in-a-specific-column-in-pandas-dataframe/49291779#49291779 , but this is likewise not clean, IMHO.

Any suggestions?


Re: fiona SQL filter

Loïc Dutrieux
 

Thanks Alan, you're right. I wrongly assumed that this was an old or at least already existing feature since it is in the online doc.

Cheers,
Loïc
________________________________________
From: main@fiona.groups.io <main@fiona.groups.io> on behalf of Alan Snow <alansnow21@gmail.com>
Sent: 03 June 2021 20:05:29
To: main@fiona.groups.io
Subject: Re: [fiona] fiona SQL filter

I think the feature you want is slated for 2.0: https://github.com/Toblerity/Fiona/pull/961<https://urldefense.com/v3/__https://github.com/Toblerity/Fiona/pull/961__;!!DOxrgLBm!UB5tJPQHin0tXeTB6cruOVmD3nIa-Dq8tSRv6xve6RsrgL5eedhcf0yBmtwj9Z5O8Cik8ns$>


Re: fiona SQL filter

Alan Snow
 

I think the feature you want is slated for 2.0: https://github.com/Toblerity/Fiona/pull/961


fiona SQL filter

Loïc Dutrieux
 

Hi everyone,

I'm trying to do some attribute filtering of a fiona feature collection using the where clause but I cannot find out the right syntax. Here's what I've tried.

Thanks for any help,
Kind regards,
Loïc


import string
import tempfile, os

import fiona
from fiona.crs import from_epsg
from shapely.geometry import Point, mapping

# Generate test data and save to tmp geopackage
gpkg = os.path.join(tempfile.gettempdir(), 'fiona_sql_test.gpkg')
sc = [Point(1,2), Point(3,4), Point(5,6)]
fc = [{'geometry': mapping(s),
'properties': {'letter': string.ascii_lowercase[idx]}}
for idx,s in enumerate(sc)]
schema = {'geometry': 'Point',
'properties': {'letter': 'str'}}
with fiona.open(gpkg, 'w', driver='GPKG', schema=schema, crs=from_epsg(4326),
layer='test') as dst:
dst.writerecords(fc)

# Filter using OGR SQL dialect
with fiona.open(gpkg, layer='test') as src:
fc = src.filter(where='"letter" = "a"')
for feat in fc:
print(feat)


Fiona 1.8.20

Sean Gillies
 

Hi all,

Fiona 1.8.20 is on PyPI today. No changes other than a relaxation of the constraint on click version. Any version of click >= 4.0 is allowed now. The wheels on PyPI include library upgrades: GDAL 3.3.0 and GEOS 3.9.1.

--
Sean Gillies


Re: Cannot install Fiona on Mac OS, GDAL installed via homebrew

jackjomarron@...
 

Thank you for your help!

On Sun, Apr 25, 2021 at 11:35 AM <camille.modeste@...> wrote:
Hi, 

I got the same issue, with pipenv, running on Python 3.9.4. Setting Python to 3.9.1 solved this issue.



--
Jack Jomarron

   


Re: Cannot install Fiona on Mac OS, GDAL installed via homebrew

camille.modeste@...
 

Hi, 

I got the same issue, with pipenv, running on Python 3.9.4. Setting Python to 3.9.1 solved this issue.


AttributeError on import; module 'fiona' has no attribute '_loading'

jackjomarron@...
 

I am trying to map on Jupyter notebook using geopandas. I am running into some issues importing geopandas through fiona. I have tried importing fiona directly as well and still receive the same error.

Along with importing fiona directly, I have tried uninstalling and reinstalling fiona and calling the _loading attribute directly.

I have checked on conda list and have gdal 3.2.2 and fiona 1.8.19 both on the conda-forge channel.

Thanks for any help!


Re: FIona 1.1.18, GDAL 3.2.1

René Buffat
 


If you install your own proj and gdal, then you probably don't want to use the wheel version of Fiona, as the wheels contain their own version of gdal. 
With the following command you should be able to compile Fiona from source using your installed gdal:

pip install Fiona --no-binary Fiona

 

Regarding the error message: gcs.csv was replaced by a sqlite file in gdal 3.0. The current Fiona wheels include still gdal 2.4. I assume you are affected by https://github.com/Toblerity/Fiona/issues/1007

where the old gdal from the wheel finds the GDAL_DATA directory of the new gdal installation. 

 

lg rene

 

 


FIona 1.1.18, GDAL 3.2.1

David Ignjić
 

I use docker file (later) where i install fiona via pip when i make 
import fiona

I get error:
ERROR 4: Unable to open EPSG support file gcs.csv.  Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.

Do i need compile fiona ?

Thanks 
                David





Docker file:
FROM python:3.8-slim-buster

ENV DEBIAN_FRONTEND noninteractive
ENV POETRY_VERSION 1.1.4
ENV PROJ_VERSION 7.2.1
ENV GDAL_VERSION 3.2.1

RUN apt-get update -yqq \
&& apt-get install -yqq build-essential wget pkg-config file libsqlite3-dev sqlite3 libtiff-dev netcat vim procps

# PROJ
RUN wget https://download.osgeo.org/proj/proj-${PROJ_VERSION}.tar.gz \
&& tar xzf proj-${PROJ_VERSION}.tar.gz \
&& cd proj-${PROJ_VERSION} \
&& ./configure --prefix=/usr --sysconfdir=/etc --without-curl \
&& make --quiet -j4 \
&& make --quiet install \
&& make --quiet clean \
&& cd .. \
&& rm -rf proj-${PROJ_VERSION} \
&& rm -f proj-${PROJ_VERSION}.tar.gz


# GDAL

RUN wget http://download.osgeo.org/gdal/${GDAL_VERSION}/gdal-${GDAL_VERSION}.tar.gz \
&& tar xzf gdal-${GDAL_VERSION}.tar.gz \
&& cd gdal-${GDAL_VERSION} \
&& ./configure --disable-static --prefix=/usr --sysconfdir=/etc --with-libkml \
&& make --quiet -j4\
&& make --quiet install \
&& make --quiet clean \
&& cd .. \
&& rm -rf gdal-${GDAL_VERSION} \
&& rm -f gdal-${GDAL_VERSION}.tar.gz


Re: Attempting to find version of GDAL installed

Orry Messer
 

Yeah, I believe this is what has happened. On the CI/CD server, a whl was used, and locally it's using the source .tar.gz.

Thanks again.


Re: Attempting to find version of GDAL installed

Sean Gillies
 

We provide wheel distributions for only a small group of platform tags and Python versions. See https://pypi.org/project/Fiona/1.8.18/#files. I suspect your CI/CD server's pip program finds a compatible wheel, but your dev machine cannot. In that case, pip on your dev machine will take the source distribution, the .tar.gz file.


On Mon, Mar 8, 2021, 12:44 AM Orry Messer <orry@...> wrote:
Thanks Sean!

Well, it appears that the libgdal shared library which fiona is linking resides in /usr/local/lib/python3.6/site-packages/fiona/.libs, rather than /usr/lib or /usr/local/lib - which leads me to believe that fiona has obtained its GDAL dependencies from PyPI. 

What I am struggling with, is that locally, on my dev machine, fiona does not seem to be getting GDAL dependencies from PyPI, where as on my CI/CD server, it is getting its dependencies and so the result of "pip install fiona" works. Do you have any idea why this may be the case?

- Orry


Re: Attempting to find version of GDAL installed

René Buffat
 

Not sure if related, but there seems to be no macos wheel for python 3.9 on pypi: https://pypi.org/project/Fiona/#files


Re: Attempting to find version of GDAL installed

Orry Messer
 

Thanks Sean!

Well, it appears that the libgdal shared library which fiona is linking resides in /usr/local/lib/python3.6/site-packages/fiona/.libs, rather than /usr/lib or /usr/local/lib - which leads me to believe that fiona has obtained its GDAL dependencies from PyPI. 

What I am struggling with, is that locally, on my dev machine, fiona does not seem to be getting GDAL dependencies from PyPI, where as on my CI/CD server, it is getting its dependencies and so the result of "pip install fiona" works. Do you have any idea why this may be the case?

- Orry


Re: Attempting to find version of GDAL installed

Sean Gillies
 

Hi Orry,

No need to apologise for asking, the fiona and GDAL software stack is complicated. 

The fiona packages on PyPI include their own copies of GDAL and its dependencies, so you may be detecting that one. One way to know for sure is to use (on Linux) the LD_DEBUG environment variable.

LD_DEBUG=all python -c "import fiona"

will dump a wall of text into your terminal and if you scan it for libgdal, you'll see the path of the shared library that fiona links.


On Thu, Mar 4, 2021, 7:27 AM Orry Messer <orry@...> wrote:

Good day,

 

I am attempting to debug a problem I have while installing Fiona within a Docker container. Apologies if my understanding is a bit shaky – I’m very new to this.

Essentially, I am trying to understand the relationship between GDAL and Fiona. It stems from a problem to do with specifying GDAL API version in a fresh install (either by using gdal-config or GDAL_CONFIG environment variable).

I currently have a working container, with Fiona installed. However, I do not understand how it got to this state.

When I import Fiona in python, 

>>> import fiona

And then run 

>>> fiona.gdal_version

I get: GDALVersion(major=2, minor=3, revision=2).

To me, it seems that GDAL is installed within this container. However, I cannot find it anywhere in this container. Is there a way to ascertain the path Fiona is pointing to?

 

Thanks,

Orry

 


Attempting to find version of GDAL installed

Orry Messer
 

Good day,

 

I am attempting to debug a problem I have while installing Fiona within a Docker container. Apologies if my understanding is a bit shaky – I’m very new to this.

Essentially, I am trying to understand the relationship between GDAL and Fiona. It stems from a problem to do with specifying GDAL API version in a fresh install (either by using gdal-config or GDAL_CONFIG environment variable).

I currently have a working container, with Fiona installed. However, I do not understand how it got to this state.

When I import Fiona in python, 

>>> import fiona

And then run 

>>> fiona.gdal_version

I get: GDALVersion(major=2, minor=3, revision=2).

To me, it seems that GDAL is installed within this container. However, I cannot find it anywhere in this container. Is there a way to ascertain the path Fiona is pointing to?

 

Thanks,

Orry

 


Re: Cannot install Fiona on Mac OS, GDAL installed via homebrew

amit@...
 

Hi Sean, 

Thanks for your reply, and the relevant link - my issue came down to simply not having a properly updated and configured SDK version, once I took care of that, it works with out issues.

If anyone else comes across the same issue, here is what I did to solve it:

1. Download and install the lates version of [XCode](https://developer.apple.com/xcode/)
2. Browse to "/Library/Developer/CommandLineTools/SDKs"
3. Ensure that the alias "MacOSX.sdk" points to the proper SDK version (mine has been pointing to "MacOSX11.1.sdk" instead of the appropriate "MacOSX10.15.sdk" - I am not sure when how or why happened exactly)
4. Install fiona


Re: Cannot install Fiona on Mac OS, GDAL installed via homebrew

Sean Gillies
 

Hi Amit,

I haven't seen this error before, but I also stopped developing fiona on macOS two years ago. High Sierra was what I was using. https://stackoverflow.com/a/61526989/159235 looks like it could be relevant.




On Mon, Feb 22, 2021 at 9:33 AM <amit@...> wrote:
Hey,

This is my first interaction with Fiona (and geo-related platforms in general), I ended up here after failing to install `geoplot` and `geopandas` using a package manager (Poetry), and finally pinned it down to issues installing Fiona.

I cannot seem to install Fiona. This is what I am currently doing:

1. Install GDAL through hombrew (`brew install gdal`)
2. Create a new venv (named `fiona_test`), using Python 3.9.0 (managed by pyenv).
3. Activate the venv.
4. run `gdal-config --version` seems to be fine -> shows `3.2.1`
5. run `pip install fiona` -> results in error.

Running macOS Catalina Version 10.15.17.

Full error on terminal is attached.

Any help would be greatly appreciated, thanks in advance!

Amit



--
Sean Gillies

1 - 20 of 107