Commit Graph

48 Commits (12a549ce6a5f5bd157a53cd12a3c0a7e079f2271)

Author SHA1 Message Date
Dessalines 12a549ce6a Addressing PR comments 2 months ago
Dessalines f3d48f2c2c
Adding some recommended fixes from nightly clippy. (#4473) 3 months ago
dullbananas f56b84615c
Move DbUrl trait impls to newtypes.rs (#4463)
* Move DbUrl trait impls to newtypes.rs

* Update utils.rs
3 months ago
dullbananas d79502dff3
Escape backslashes in fuzzy_search (#4462)
* Escape backslashes in fuzzy_search

* Update utils.rs
3 months ago
Nutomic ffcf415cac
Dont log db url on connection error (fixes #4453) (#4456)
* Dont log db url on connection error (fixes #4453)

* remove format
3 months ago
dullbananas 677d54ae57
Allow better query plans (#4424)
* Update utils.rs

* Create bind_if_some.rs

* limit connection age

* Delete crates/db_schema/src/utils/bind_if_some.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs

* Update utils.rs
3 months ago
dullbananas d8f9e8a64c
Post view: move cursor pagination to separate library, add backward pagination to PostQuery (#4320)
* stuff

* stuff

* crates.io

* Update up.sql

* Rerun federation tests

* Update post_view.rs

* Update post_view.rs

* Update up.sql

* Update utils.rs

* Fix precision loss

* Update up.sql

* Update down.sql

* remove unwrap

* Update post_view.rs

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
4 months ago
dullbananas 759f6d8a9a
Better query plan viewing experience (#4285)
* stuff

* stuff including batch_upsert function

* stuff

* do things

* stuff

* different timestamps

* stuff

* Revert changes to comment.rs

* Update comment.rs

* Update comment.rs

* Update post_view.rs

* Update utils.rs

* Update up.sql

* Update up.sql

* Update down.sql

* Update up.sql

* Update main.rs

* use anyhow macro

* replace get(0) with first()

* as_slice

* Update series.rs

* Update db_perf.sh

* Update and rename crates/db_schema/src/utils/series.rs to crates/db_perf/src/series.rs

* Update utils.rs

* Update main.rs

* Update main.rs

* Update .woodpecker.yml

* fmt main.rs

* Update .woodpecker.yml

* Instance::delete at end

* Update main.rs

* Update Cargo.toml

---------

Co-authored-by: Nutomic <me@nutomic.com>
4 months ago
Nutomic 7d9b59c467
Reduce default db pool size to 30, remove db timeout (ref #4282) (#4301)
* Reduce default db pool size to 30 (ref #4282)

* remove db timeout
5 months ago
ALEX11BR a5289dd4cf
Use `pretty_assertions` for assertions (#4347)
* Use `pretty_assertions` for assertions

* fixed ordering of `use`s

* ci

---------

Co-authored-by: Felix Ableitner <me@nutomic.com>
5 months ago
Kroese 2d4037ba61
Extend sitemap span (#4231)
* Extend sitemap span

* Keep cargo_fmt happy

* Add FETCH_LIMIT_SITEMAP

* Use FETCH_LIMIT_SITEMAP

* Keep cargo_fmt happy

* Update utils.rs

* Use SITEMAP_DAYS

* Keep cargo_fmt happy

* Sitemap

* Keep cargo_fmt happy

* Sitemap

* Sitemap

* Increase to 31 days
5 months ago
dullbananas d95df3a46f
Update diesel-async (#4203)
* Update diesel-async

* Fix
6 months ago
Dessalines 6e4278b1c9
Setting rank column defaults to a small number. Fixes #4178 (#4191)
* Setting rank column defaults to a small number. Fixes #4178

* Fixing tests.
6 months ago
Nutomic 08739e2925
Move usage of env::var to lemmy_utils, simplify db init (ref #4095) (#4108) 7 months ago
Nutomic 332e698336
Cleanup public api (#4047)
* Convert PersonSortType to purely internal

* Remove hot rank and other db optimizations from public api
7 months ago
phiresky 375d9a2a3c
Persistent, performant, reliable federation queue (#3605)
* persistent activity queue

* fixes

* fixes

* make federation workers function callable from outside

* log federation instances

* dead instance detection not needed here

* taplo fmt

* split federate bin/lib

* minor fix

* better logging

* log

* create struct to hold cancellable task for readability

* use boxfuture for readability

* reset submodule

* fix

* fix lint

* swap

* remove json column, use separate array columns instead

* some review comments

* make worker a struct for readability

* minor readability

* add local filter to community follower view

* remove separate lemmy_federate entry point

* fix remaining duration

* address review comments mostly

* fix lint

* upgrade actitypub-fed to simpler interface

* fix sql format

* increase delays a bit

* fixes after merge

* remove selectable

* fix instance selectable

* add comment

* start federation based on latest id at the time

* rename federate process args

* dead instances in one query

* filter follow+report activities by local

* remove synchronous federation

remove activity sender queue

* lint

* fix federation tests by waiting for results to change

* fix fed test

* fix comment report

* wait some more

* Apply suggestions from code review

Co-authored-by: SorteKanin <sortekanin@gmail.com>

* fix most remaining tests

* wait until private messages

* fix community tests

* fix community tests

* move arg parse

* use instance_id instead of domain in federation_queue_state table

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
Co-authored-by: SorteKanin <sortekanin@gmail.com>
8 months ago
Dessalines 9785b20843
Adding a scaled sort, to boost smaller communities. (#3907)
* Adding a scaled sort, to boost smaller communities.

- Previously referred to as *best* .
- Fixes #3622

* Fixing scheduled task update.

* Converting hot_rank integers to floats.

* Altering hot_rank psql function to default to zero after a week.

* Setting scaled_rank to zero, where hot_rank is zero.

* Adding image_upload table.
9 months ago
dullbananas 15930cbf4d
Use Queryable instead of JoinView (#3917)
* Update utils.rs

* Update traits.rs

* Update comment_report_view.rs

* Update comment_view.rs

* Update local_user_view.rs

* Update post_report_view.rs

* Update post_view.rs

* Update private_message_report_view.rs

* Update private_message_view.rs

* Update registration_application_view.rs

* Update site_view.rs

* Update structs.rs

* Update comment_reply_view.rs

* Update community_block_view.rs

* Update community_follower_view.rs

* Update community_moderator_view.rs

* Update community_person_ban_view.rs

* Update community_person_ban_view.rs

* Update community_view.rs

* Update person_block_view.rs

* Update person_mention_view.rs

* Update person_view.rs

* Update structs.rs

* Update admin_purge_comment_view.rs

* Update admin_purge_community_view.rs

* Update admin_purge_person_view.rs

* Update admin_purge_post_view.rs

* Update mod_add_community_view.rs

* Update mod_add_view.rs

* Update mod_ban_from_community_view.rs

* Update mod_ban_view.rs

* Update mod_feature_post_view.rs

* Update mod_hide_community_view.rs

* Update mod_lock_post_view.rs

* Update mod_remove_comment_view.rs

* Update mod_remove_community_view.rs

* Update mod_remove_post_view.rs

* Update mod_transfer_community_view.rs

* Update structs.rs

* Update utils.rs

* Update private_message_view.rs

* Update comment_report_view.rs

* Update registration_application_view.rs

* Update utils.rs

* fix

* fix db_views

* fix

* Update comment_view.rs
9 months ago
phiresky 514f2222e0
Fix time zone handling (#3496)
* convert naive time to utc time

* compounding fixes

* cargo fmt

* fix the rest

* fix down migration

* fix migrations

* fix after merge

* clippy fix

* ap-fed 0.5.0

---------

Co-authored-by: Nutomic <me@nutomic.com>
9 months ago
dullbananas 9a5a13c734
Use same table join code for both read and list functions (#3663)
* Try stuff

* Revert "Try stuff"

This reverts commit 3da5f83a8b.

* Revert "Revert "Try stuff""

This reverts commit 178bd43cac.

* Revert "Revert "Revert "Try stuff"""

This reverts commit b9f9a2316e.

* Revert "Revert "Revert "Revert "Try stuff""""

This reverts commit ccd498dd72.

* Try more stuff

* Add queries function

* Simplify queries function

* Move aliases to db_schema

* Revert "Move aliases to db_schema"

This reverts commit 69afed05c1.

* Add ReadFuture and ListFuture

* Refactor queries function and add Queries struct

* Box futures in Queries::new

* Use from_tuple

* Add comment_view::queries and improve comment_report_view::queries

* Add local_user_view::queries

* Add post_report_view::queries

* Ad post_view::queries

* Add private_message_report_view::queries

* private_message_view, registration_application_view

* Use 'a in BoxedQuery

* comment_reply_view, community_view

* Change aliases to inline module

* person_mention_view

* person_view

* Use separate community_person_ban joins instead of including boolean literal in join-on clause

* Fix comment_view

* rerun ci
10 months ago
Nutomic 3471f3533c
Sanitize html (#3708)
* HTML sanitization in apub code

* Sanitize API inputs

* fmt

* Dont allow html a, img tags

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
10 months ago
Pavlos Smith c890797b37
Add controversial ranking (#3205)
* Added controversy rank property to posts and comments, and ability to sort by it

* Triggers instead of schedules tasks, integer -> double, TODO: comments don't seem to get updated with floats, divide SortTypes

* Created PersonSortType

* PersonSortType::MostComments case

* Removed unused PartialOrd trait

* Added new person sort type mappings

* SortType -> PersonSortType

* fixes

* cargo fmt

* fixes after merge with main

* Fixed bug in controversy rank trigger, removed TopX sorts from PersonSortType and added CommentScore instead

* Uncovered enum case

* clippy

* reset translation changes

* translations

* translations

* Added additional hot ordering on controversial posts and comments

* featured local and featured community added to controversy rank index, additional order_by removed (?), added post_score and post_count to PersonSortType

* Woodpecker rerun

* cargo fmt

* woodpecker rerun

* fixed controversy_rank order

* fix

* Readded migration as latest, removed second update statement for setting controversy rank
10 months ago
cetra3 9256895635
Cache & Optimize Woodpecker CI (#3450)
* Try using drone cache plugin

* Try another path

* Include volume

* Fix formatting

* Include fmt

* Exclude cargo dir from prettier

* Don't override cargo

* Just do check

* Add cache key

* Use different cache plugin

* Add clippy

* Try minio

* Add quotes

* Try adding secrets

* Try again

* Again

* Use correct secret formation

* Add back clippy

* Use secret for the root bucket name

* Try drone cache instead

* Add region

* Add path-style option

* Include cargo clippy

* Include everything again

* Fix formatting

* Don't run clippy twice

* Add `allow` statements for tests to pass

* Adjust endpoint to be a secret

* Fix prettier

* Merge & fix tests

* Try to restart the woodpecker test

* Change the ENV var name

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
10 months ago
dullbananas 1d38aad9d3
Make functions work with both connection and pool (#3420)
* a lot

* merge

* Fix stuff broken by merge

* Get rid of repetitive `&mut *context.conn().await?`

* Add blank lines under each line with `conn =`

* Fix style mistakes (partial)

* Revert "Fix style mistakes (partial)"

This reverts commit 48a033b87f.

* Revert "Add blank lines under each line with `conn =`"

This reverts commit 773a6d3beb.

* Revert "Get rid of repetitive `&mut *context.conn().await?`"

This reverts commit d2c6263ea1.

* Use DbConn for CaptchaAnswer methods

* DbConn trait

* Remove more `&mut *`

* Fix stuff

* Re-run CI

* try to make ci start

* fix

* fix

* Fix api_common::utils

* Fix apub::activities::block

* Fix apub::api::resolve_object

* Fix some things

* Revert "Fix some things"

This reverts commit 2bf8574bc8.

* Revert "Fix apub::api::resolve_object"

This reverts commit 3e4059aabb.

* Revert "Fix apub::activities::block"

This reverts commit 3b02389abd.

* Revert "Fix api_common::utils"

This reverts commit 7dc73de613.

* Revert "Revert "Fix api_common::utils""

This reverts commit f740f115e5.

* Revert "Revert "Fix apub::activities::block""

This reverts commit 2ee206af7c.

* Revert "Revert "Fix apub::api::resolve_object""

This reverts commit 96ed8bf2e9.

* Fix fetch_local_site_data

* Fix get_comment_parent_creator

* Remove unused perma deleted text

* Fix routes::feeds

* Fix lib.rs

* Update lib.rs

* rerun ci

* Attempt to create custom GetConn and RunQueryDsl traits

* Start over

* Add GetConn trait

* aaaa

* Revert "aaaa"

This reverts commit acc9ca1aed.

* Revert "Revert "aaaa""

This reverts commit 443a2a00a5.

* still aaaaaaaaaaaaa

* Return to earlier thing

Revert "Add GetConn trait"

This reverts commit ab4e94aea5.

* Try to use DbPool enum

* Revert "Try to use DbPool enum"

This reverts commit e4d1712646.

* DbConn and DbPool enums (db_schema only fails to compile for tests)

* fmt

* Make functions take `&mut DbPool<'_>` and make db_schema tests compile

* Add try_join_with_pool macro and run fix-clippy on more crates

* Fix some errors

* I did it

* Remove function variants that take connection

* rerun ci

* rerun ci

* rerun ci
10 months ago
Nutomic 93225e5ddf
Error enum fixed (#3487)
* Create error type enum

* Replace magic string slices with LemmyErrorTypes

* Remove unused enum

* Add rename snake case to error enum

* Rename functions

* clippy

* Fix merge errors

* Serialize in PascalCase instead of snake_case

* Revert src/lib

* Add serialization tests

* Update translations

* Fix compilation error in test

* Fix another compilation error

* Add code for generating typescript types

* Various fixes to avoid breaking api

* impl From<LemmyErrorType> for LemmyError

* with_lemmy_type

* trigger ci

---------

Co-authored-by: SleeplessOne1917 <abias1122@gmail.com>
10 months ago
Dessalines 8bcf2ea7c9
Fixing clippy. (#3471) 11 months ago
c-andy-candies f5209fffc1
Feature add three six and nine months options backend (#3226)
Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
11 months ago
Felix Ableitner ce0cf0e41b Remove DELETED_REPLACEMENT_URL 11 months ago
Sander Saarend 6b28f8c616 Add support for sslmode=require for diesel-async DB connections (#3189) 11 months ago
Vijay Ramesh 4db65c191c after 30 days post deletion, replace comment.content and post.body with 'Permanently Deleted' 11 months ago
Ivo Barros 4d160461ce
Feature add hours as sorting options backend (#3161)
* add hours as sorting options

* ran cargo +nightly fmt

* woodpicker trigger
11 months ago
Carlrs 24323e17b2
update diesel to 2.1.0 and diesel-async to 0.3.0 (issue #2882) (#2895)
* update diesel to 2.1.0 and diesel-async to 0.3.0 (issue #2882)

* get rid of deprecated RawValue type alias usage, explicitly import diesel types instead of using wildcard import

* Update diesel-derive-enum and diesel-derive-newtype libraries

* Revert manual changes to schema.rs

* Update to newer diesel-cli image in CI, run fix-clippy before schema comparison, update schema to one generated with diesel-cli 2.1.0 and clippyfixed

* Remove wildcard import lint, fix diesel schema pipeline

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
12 months ago
Nutomic 6f513793cb
Activitypub crate rewrite (#2782)
* update activitypub-federation crate to 0.4.0

* fixes

* apub compiles!

* everything compiling!

* almost done, federated follow failing

* some test fixes

* use release

* add code back in
1 year ago
Dessalines 66b0ddbbc5
Adding deadpool timeouts. (#2775) 1 year ago
Dessalines 209c8a9185
Switch from bb8 to deadpool. Fixes #2765 (#2768)
* Switch from bb8 to deadpool. Fixes #2765

* Remove unecessary deadpool dependency.

* Ignoring nodeinfo test.
1 year ago
Nutomic 62663a9f2e
Implement separate mod activities for feature, lock post (#2716)
* Implement separate mod activities for feature, lock post

Also includes collection for featured posts. Later we also need
to do the same for Comment.distinguished

* some changes

---------

Co-authored-by: Dessalines <dessalines@users.noreply.github.com>
1 year ago
Nutomic 2891856b48
Fix docker federation setup (#2706) 1 year ago
Dessalines a610211557
Fixing `.drone.yml` (#2677)
* Try to fix docker/drone plugin

* Trying to use one rust image.

* Fixing drone 1.

* Fixing drone 2.

* Add drone notify.

* Fixing drone 3.

* Fixing drone 4.

* Fix clippy.

* Remove uninlined format lint.

* Combine all cargo tasks

* Fixing drone 5.

* Fixing drone 6.

* Fixing drone 7.

* Fixing drone 8.

* Fixing drone 9.

* Fixing drone 10.

* Fixing drone 12.

* Fixing drone 13.

* Fixing drone 14.

* Fixing drone 15.

* Fixing drone 16.

* Fixing drone 17.

* Fixing drone 18.

* Fixing drone 19.

* Fixing drone 20.

* Fixing drone 21.

* Fixing drone 22.

* Fixing drone 23.

* Fixing drone 24.

* Fixing drone 25.

* Fixing drone 26.

* Fixing drone 27.

* Fixing drone 28.

* Fixing drone 29.

* Fixing drone 30.

* Fixing drone 31.

* Fixing drone 32.

* Fixing drone 33.

* Fixing drone 34.

* Fixing drone 35.

* Fixing drone 36.

* Fixing drone 37.

* Fixing drone 38.

* Fixing drone 39.

* Fixing drone 40.

* Fixing drone 41.

* Fixing drone 43.

* Fixing drone 44.

* Fixing drone 45.

* Last cleanup.

* Fixing drone 46.

* Separate ci steps (#2679)

* separate ci steps

* fix 1

* add comments

* dont add rustfmt explicitly

* Revert "dont add rustfmt explicitly"

This reverts commit 358ce3302a134b7ac88d90a854079356995e9725.

* dont use all features for tests

---------

Co-authored-by: Nutomic <me@nutomic.com>
1 year ago
Dessalines 70c549dad8
Speeding up comment-ltree migration, fixing index creation. Fixes #2664 (#2670)
* Speeding up comment-ltree migration, fixing index creation. Fixes #2664

* Adding some logging lines, fixing for missing posts.

* Adding more postgres config
1 year ago
Nutomic 6f3bf4634b
Various pedantic clippy fixes (#2568)
* Various pedantic clippy fixes

* more clippy pedantic fixes

* try to fix ci

* add fix clippy script, use rust 1.65

* fix clippy
2 years ago
Dessalines 5d837780f5
Add diesel_async, get rid of blocking function (#2510)
* Moving settings to Database.

- Moves many settings into the database. Fixes #2285
- Adds a local_site and instance table. Fixes #2365 . Fixes #2368
- Separates SQL update an insert forms, to avoid runtime errors.
- Adds TypedBuilder to all the SQL forms, instead of default.

* Fix weird clippy issue.

* Removing extra lines.

* Some fixes from suggestions.

* Fixing apub tests.

* Using instance creation helper function.

* Move forms to their own line.

* Trying to fix local_site_data, still broken.

* Testing out async

* Testing out async 2

* Fixing federation tests.

* Trying to fix check features 1.

* Starting on adding diesel async. 1/4th done.

* Added async to views and schema.

* Adding some more async

* Compiling now.

* Added diesel async. Fixes #2465

* Running clippy --fix

* Trying to fix cargo test on drone.

* Trying new muslrust.

* Trying a custom dns

* Trying a custom dns 2

* Trying a custom dns 3

* Trying a custom dns 4

* Trying a custom dns 5

* Trying a custom dns 6

* Trying a custom dns 7

* Addressing PR comments.

* Adding check_apub to all verify functions.

* Reverting back drone.

* Fixing merge

* Fix docker images.

* Adding missing discussion_languages.

* Trying to fix federation tests.

* Fix site setup user creation.

* Fix clippy

* Fix clippy 2

* Test api faster

* Try to fix 1

* Try to fix 2

* What are these lines about

* Trying to fix 3

* Moving federation test back to top.

* Remove logging cat.
2 years ago
Dessalines 235cc8b228
Moving settings to Database. (#2492)
* Moving settings to Database.

- Moves many settings into the database. Fixes #2285
- Adds a local_site and instance table. Fixes #2365 . Fixes #2368
- Separates SQL update an insert forms, to avoid runtime errors.
- Adds TypedBuilder to all the SQL forms, instead of default.

* Fix weird clippy issue.

* Removing extra lines.

* Some fixes from suggestions.

* Fixing apub tests.

* Using instance creation helper function.

* Move forms to their own line.

* Trying to fix local_site_data, still broken.

* Fixing federation tests.

* Trying to fix check features 1.

* Addressing PR comments.

* Adding check_apub to all verify functions.
2 years ago
Dessalines c9f1407429
Diesel 2.0.0 upgrade (#2452)
* Initial commit to bump diesel to 2.0.0-rc.0 and see what happens

* Add chrono feature from diesel

* db_schema crate is close to building?

* Upgrade diesel-derive-newtype

* Mostly modifying references to connections to be mutable ones; also used
new way to do migrations as suggested by the migration guide; a lot more
compiles now, though I can't figure out this tricky ToSql issue at the
moment

* Running clippy --fix

* Trying to fix drone clippy 1

* Fix clippy

* Upgrade clux-musl

* Trying to fix drone clippy 2

* Trying to fix drone clippy 3

* Trying to fix drone clippy 5

* Adding diesel table aliases, removing sql view hack. Fixes #2101

Co-authored-by: Steven Chu <stevenc1@gmail.com>
Co-authored-by: Nutomic <me@nutomic.com>
2 years ago
Dessalines 9c3efe32e7
First pass at adding comment trees. (#2362)
* First pass at adding comment trees.

- Extracted comment replies into its own table.
- Added ltree column to comment
- Added parent_id param to GetComments to fetch a tree branch
- No paging / limiting yet

* Adding child_count to comment_aggregates.

* Adding parent comment update counts

* Fix unit tests.

* Comment tree paging mostly done.

* Fix clippy

* Fix drone tests wrong postgres version.

* Fix unit tests.

* Add back in delete in unit test.

* Add postgres upgrade script.

* Fixing some PR comments.

* Move update ltree into Comment::create

* Updating based on comments.

* Fix send soft fail.
2 years ago
Dessalines ff026dc3ff
Adding check for requests with no id or name, adding max limit. (#2265)
* Adding check for requests with no id or name, adding max limit.

* Consolidating a few functions.

* Fix page min

* Adding more websocket rate limits.

* Add check to GetCommunity

* Use a default message rate limit check.

* Adding a page and limit checker

* Fix clippy

* Fix clippy again

* Adding check for requests with no id or name, adding max limit.

* Consolidating a few functions.

* Fix page min

* Adding more websocket rate limits.

* Add check to GetCommunity

* Use a default message rate limit check.

* Adding a page and limit checker

* Fix clippy

* Fix clippy again

* Fix limit request.

* Move checks to inside limit_and_offset

* Fixing API tests.

* Change NotFound diesel errors to QueryBuilderError
2 years ago
Nutomic a2a594b763
Extract Activitypub logic into separate library (#2288)
* Create example for apub lib

* some rewriting of apub lib

* Add LocalInstance struct for apub lib to avoid using Lemmy Settings

* Move ActorType trait to lemmy_apub, because its not needed in library

* Use reqwest_retry instead of custom impl, dont specify timeout on every send()

* Some improvements to example

* Moved inbox handling to library

* bug fixes

* Move context and serde helpers into library

* wip: example changes

* Add lemmy_utils feature to build only LemmyError

* Rename to activitypub_federation

* Remove lemmy_utils dep from activitypub_federation using generic error type

* Finish activitypub example

* Cleanup and fix tests

* Reorganize library files

* Remove ApubObject.to_tombstone()

* Extract activitypub library into separate git repository
2 years ago
Nutomic 3053e14be7
Derive default for api request structs, move type enums (#2245)
* Derive default for api request structs, move type enums

* Simplify api by using enum types directly, instead of string

* Add default and clone for most api structs
2 years ago
Nutomic 3aa3d75a1e
Add cargo feature for building lemmy_api_common with mininum deps (#2243) 2 years ago